1.

Why is dataset shift a problem area? Explain briefly about some ideas that may address those problems in data science?

Answer»

Response: 

As we understand, dataset shift is a problem when we put our model from development environment to production environment. These of course are classified into VARIOUS types DEPENDING on whether there is a shift that occurs later between an independent and target VARIABLE, or within independent variables or with the target variable only. 

The causes for dataset shift can be due to the following factors: 

  1. Production model is no longer fit for purpose because of changes due to data distribution, variation in data parameters/features etc. 
  2. May be difficult to detect if there is any such dataset shift 
  3. There is an inherent need to monitor models that are in production regularly to ensure model performance does not degrade 
  4. Changes in behaviour of model features may be sequential, gradual or ad-hoc (depending on data quality and how data changes over some time) 
  5. Increased model maintenance 

Following attempts can be taken to address these issues at hand. 

  1. Re-fit or upgrade the model periodically based on a certain frequency by checking model performance or ACCURACY (e.g. checking Precision / Recall for a certain scenario against new data for few weeks) 
  2. Keep monitoring distribution of independent variables in the dataset in the production environment. 
  3. Keep assessing the model performance periodically. 
  4. Weight the data 
  5. Learn the change in features in the dataset. 


Discussion

No Comment Found