Machine Learning Prediction

this use case is a customized time series forecasting version of making a cnn prediction model from the tensorflow website the code is dived into multiple segments and expected output is posted to make sure there are no errors, making it easier to debug the complete tutorial is available on the tensorflow https //www tensorflow\ org/tutorials/structured data/time series website setup and data if you run the script below, you see an output of a dataframe (table) feature engineering / data standardization now with the quick setup done, we can do some feature engineering / extraction and data manipulation which will help the model there are plenty of options when it comes to feature engineering and it is case by case basis, but for this example, it will be kept almost exactly the same as tensorflow plot of entire data set plot of initial part of data set cleanup most of this section is highly dependent on the kind of data being used this data is of the weather, and various parameters are used to predict the temperature so most of this cleanup is specific to these kinds of datasets, and maybe completely irrelevant to other kinds of datasets wind velocity as the minimum value of wind velocity (wv m/s) and max wv m/s is 9999, it seems incorrect, as we already have wind direction, let's correct this minimum note this may not apply to all kinds of datasets changing wind variables to see why we need to change the wind variables, lets plot them first original wind variables not to get into too many details, but ideally 0 degrees and 360 degrees should be next to each other, which they aren't and it should not be a sharp change at 0 degree mark also, direction of the wind does not matter if its velocity is 0 (no wind) so let's make some changes to the dataframe and make wind velocity a vector, rather than scalar with degrees \### remove these columns from the data frame and replace them as vectors modified wind variables the model interprets this kind of wind variables better than it does with velocity and direction quick summary we took our desired number of features/columns from the original data set modified the data such that there is no 9999 as the minimum value in the wind velocity changed wind variables from wind velocity + wind direction (degrees) into > wind velocity vector in x & y components more cleanup timestamp, date time in string, time in seconds is not really that useful for the model, so we can make features instead we can convert date time to seconds, and then to sin & cos of years and days this will help simplify input to the model and make it more useful to identify periodicity in the data preparing data for the model all the necessary cleanups and some modifications to the data are done, now let's prepare the data for the model split the data split the data into training, validation and testing with ratios training 70% , validation 20%, testing 10% normalize the data use the simple mean to normalize the data quick view of how the data looks after normalization functions for making a moving window generator and making plots this process is a methodology you can also use other methods see the tensorflow https //www tensorflow\ org/tutorials/structured data/time series website for more information extremely simplistic model to truly understand what is going on, let's define a very simple model, which uses the variables that are currently in place, and makes a prediction for the next hour to make the model more interesting, repeat this window for 24 hrs instead just 1 hr three batches of the basic model actual cnn model cnn predictions performance comparison performance comparison save the model saving a model in keras is simple