Handling Missing Data

In a survey, sometimes not all questions on the questionnaire are answered or filled in completely. This condition causes missing data. Missing data will result in inaccurate parameter estimation due to reduced data size. Missing data can be caused by a number of things, namely the refusal of the respondent to answer several questions because the questions are personal/confidential; errors during data collection, for example questions were missed so no answers were obtained; besides that missing data can also be caused by errors during data entry.

Missing data handling can be done by deleting the incomplete data. If the incomplete data is relatively small, compared to the entire data, deleting incomplete data is a reasonable approach. But generally this will give a valid conclusion, only when missing data is random, in the sense that the probability of a response does not depend on the values of the observed or missing data.

The right solution so that the sample size is not reduced If there is missing data is to do imputation. Imputation is the process of filling in or replacing missing values in a dataset with possible values based on the information obtained in the dataset. Some of the imputation methods that can be used are:

Replace missing data with a constant value

The most common and easiest imputation method to use is to replace missing data with an average value or with a mode depending on the type of data. For numerical data, a method of replacing missing data with an average value is used, while for categorical data, a method of replacing missing data with a mode value is used. This method produces an average estimator for the variables that will be the same as the imputed value. The advantage of this method is to fill in the missing data values with expected values that have a relatively high level of stability. Meanwhile, the weakness is that the variance obtained by this method does not match the actual data and the correlation between variables can provide misleading information. This method will cause an estimated error that is always lower than the actual (underestimated error). So this method is not recommended for use.

Hot Deck Method

This method is a refinement of the method of replacing missing data with average values, especially in estimating standard errors that are underestimated. Before using this method, the data is first sorted based on the variables that are assessed to be related to the variables that have missing data items. Individuals who are in the same cluster are placed in the same file. The process of this method is to first determine the value of the selected demographics or other variables. Missing values will be replaced with the previous data values after the data is arranged in order and its identifying values (demographics) are assigned. The weakness of the Hot Deck is that if there is a lot of missing data, the value will be filled in repeatedly, so the estimation will be biased.

Regression Method

In this method, missing data is obtained by making predictions using regression. There are many types of regression models that can be used in imputed regression, for example linear regression, logistic regression. Suppose the Y variable is obtained from data with missing data and the Z variable is obtained from complete data. If Y and Z are related, we can predict the value of Y. Suppose that Y and Z are related to the model Y = f(Z) + ε, where f is a function, such as f(Z) = β0 +β1z, and ε is a random variable. If it is known that the function = f () then a value will be obtained to input the missing data yj.

EM Method (Expectation Maximisation)

The Expectation Maximization method is a method used to estimate unknown population parameters. The EM method uses an iterative procedure to calculate sufficient statistics and estimate parameters. The EM algorithm is formulated to create a model that fits the maximum likelihood (ML). In the EM algorithm, it is necessary to fill in the missing values first, then find the maximum possible. This process is repeated until it produces a parameter buffer.

Multiple Imputation Method

In this method, each missing data is not estimated through dummy values, but is obtained by representing a random sample of missing values. In this method m complete datasets are obtained, then each data set is analyzed using the complete data method. Then, the results obtained from these m datasets are combined. This process produces statistically valid inferences that reflect the uncertainty due to the missing values.