Trends prediction is a component of big data that countless fields utilize. It can make patterns trends like retail consumer sales, wellness, climate change, and energy consumption apparent.
The duration of predictions that companies use aren’t limited to five or 10 years. Some large companies look forward 50 to 100 years with the data they have today to see how viable certain subjects will be. They can do so because of the large amount of data that they have.

Take ground water evaluation as an example of one such field that utilizes data science. You have the conditions underground, which drives the way that water changes each year. These conditions can influence how much sulfur, iron, magnesium, and other minerals are found in the water, which in turn affects the quality of water derived from the grind for drinking.

Long-term business trends are no different when it comes to prediction models. The amount of data they must consider can be just as extensive.

Data models exist to put reason behind things that seem out of the ordinary. Things like an increase in more natural disaster might make sense because of global warming, but sudden shifts in climate may seem impossible to explain without considering all the factors behind it. Even things like the correlation between childhood obesity and more healthcare problems seems straightforward, but what if a treatment that decreases childhood obesity is created in the next five years to make the problem of healthcare substantially less pressing?

That means one emerging challenge with data sceince is ensuring that the “shell life” of models can be determined. This creates an actionable date that a model should again be examined for viability.

The job of determining trend viability was considerably simpler when trends could be broken down into short-, medium-, and long-term models. This allowed reports to be made weekly, quarterly, or annually, which in turn made it possible to determine how long the forecast would be viable.

The challenge with big data trends is that they encompass such a long period of time and so much data that these types of old reporting styles are insufficient to properly assess the current variables surrounding a model.

Consider companies that conduct construction for local cities. These periods can’t account for an influx in young population, nor can they assess what type of homes future populations will want. They can’t even determine what spread of young, middle-aged, old, or elderly population will be living in a city between 50 to 100 years from now.

These are just a few of the types of problems that professionals in the field of data sceince are trying to tackle. Adjusting for the hard-to-predict trends will make prediction models more accurate in the long-term, which in turn makes big data models able to give more trustworthy prediction as their time frame increases.

That leaves companies utilizing these types of data models to decide what the expiration date on their data is. Determining this single piece of data will make the models companies use more viable in the long term based on the data they capture.

Determinations like this will be the next big advance in long-term data modeling.

Introduction

SQL Server provides an easy way to integrate custom user-defined aggregated function implemented in .NET project. While a simple function with internal flat type variables  (i.e. blittable: Int, Double) requires few easy steps, a function with  a more complex structures, such as internal classes or arrays, requires extra care. While in the first case, the framework handles all the necessary steps to serialize and deserialize the structure, when dealing with internal classes we are required to make sure that we are searlizing and desearializng out internal objects correctly. In the following example we will explore a simple way to create a user-defined aggregate function using Visual Studio 2013 and .NET 4.5

Step 1

Open the visual studio 2013 and create “SQL Database Project”, by selecting New from the File menu and then New Project. A new dialog window will be opened. Enter the name of the project (“AggregateFunctionProject”) and its location on the disk and press enter

Then select New Item by using the right click on the new project (“AggregateFunctionProject”)
that appears in the Solution Explorer window. A new dialog will appear and then you need to select SQL CLR C# from the left list and SQL CLR C# Aggregate from the right list.

A new file will appear containing a template function of an aggregated function

Step 2

An aggregated function is created by defining a struct with SqlUserDefinedAggregate attribute.  The attribute defines the following

The struct itself must contain at least the following methods:

• Init: This is called when a new group of values is going to be handled using an instance of the structure
• Accumulate: Each value is passed to the Accumulate method which is responsible for making the necessary calculations, etc.
• Merge: This method is used when the original set of values is divided into several independent groups and after accumulating the group specific values, the group is merged to another group.
• Terminate: And finally when all values have been handled, Terminate returns the result.

Each of these four functions should be implemented: Init will initialise the variables. Accumulate will be used to acquire the data and calculate the value iteratively. Merge will be used to merge several sub groups into one single group and Terminate will return the final value for each group. Not in all cases the calculation can be done iteratively during the process of the Accumulate function. Sometimes we can only acquire the data in the Accumulation and the actual calculation is done in the Merge or/and in the Terminate method. In these cases, the acquisition of the data requires an internal structures that are not blittable.

Step 3

For example suppose we would like fit a linear curve to some data points in few different experiments. The data would look like this:

Our aim is to calculate the best linear fit for each one of the experiments. To do that we would use the following script:

SELECT ExperimentId, dbo.curveFitting(datapoint1,datapoint2,’a’) ‘a’, dbo.curveFitting(datapoint1,datapoint2,’b’) ‘b’ FROM data

GROUP BY ExperimentId

 The script group each one of the experiment and calculate the a parameter and the b parameter separately. The curveFitting function gets 3 input parameters: datapoint1, datapoint2 and the type of output (‘a’ for the a parameter and ‘b’ for the b parameter in the Y=aX + b linear curve equation)

To implement the curve fitting function we will use the following script:

//------------------------------------------------------------------------------
// <copyright file="CSSqlAggregate.cs" company="Meaningo">
//  All rights reserved.
// </copyright>
//------------------------------------------------------------------------------
using System;
using System.Data;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using System.Runtime.InteropServices;
using DatabaseProject2;


[SqlUserDefinedAggregate(Format.UserDefined,
        MaxByteSize = -1,
        Name = "InPlayIPI", IsInvariantToDuplicates = false,
        IsInvariantToNulls = true, IsInvariantToOrder = true,
        IsNullIfEmpty = true)]
[Serializable]
[StructLayout(LayoutKind.Sequential)]
public struct inPlayIPI : IBinarySerialize
{
    public void Init()
    {

        curve_fitting = new CurveFitting();
        num = 0;
    }

    public void Accumulate(SqlDouble dataPoint1, SqlInt32 dataPoint2, SqlString sqlOutputParameter)
    {
        string output = (string)sqlOutput;
        curve_fitting.Accumualte((double)dataPoint1, (double)dataPoint2);
        switch (sqlOutputParameter)
        {
            case "a": num = 1;
                break;
            case "b": num = 2;
                break;
            default:
                break;
        }
    }

    public void Merge(inPlayIPI Group)
    {
        this.curve_fitting.data_points.AddRange(Group.curve_fitting.data_points);
    }

    public SqlDouble Terminate()
    {
        try
        {
            
            Tuple<double, double, double> parameters = curve_fitting.FitCurve();

            switch (num)
            {
                case 1:
                    return new SqlDouble(parameters.Item1);
                case 2:
                    return new SqlDouble(parameters.Item2);
                default:
                    return new SqlDouble(0);
            }
        }
        catch (Exception e)
        {
            return new SqlDouble(0);
        }

    }

    // Our private variables  
    private CurveFitting curve_fitting;
    private int num;
}

Step 4

now the final step is to complete the two serialization functions write() and read(). In the write function we need to serialize the curveFitting class, variable after a variable in the same order that we will later read it using the read() function.

    public void Read(System.IO.BinaryReader r)
    {
        num = (Int32)r.ReadInt32();
        for (int i = 0; i <= itemCount2 - 1; i++)
        {
            this.curve_fitting.data_points.Add(new Tuple<double,double>((double)r.ReadDouble(), (double)r.ReadDouble()));
        }
    }

    public void Write(System.IO.BinaryWriter w)
    {
        w.Write(num);
        foreach(var item in curve_fitting.data_points)
        {
            w.Write(item.Item1);
            w.Write(item.Item2);
        }
    }

Prediction models can be crucial statistical tools to help companies make informed decisions. These models are basically ways of predicting events that will happen in the future based on patterns of the past through statistics. Prediction models allow those who are anticipating or considering change to become informed of the most likely result of said change. It also allows companies to more accurately make budgeting decisions as a predictive model can anticipate the potential revenues that the a product or app will continue to make.
For an example of the possible application of prediction models within the decision making of a company consider a predictive model which factors in the ad revenue that an app has made, how many users view the app and how long they tend to use it. This predictive model could favor in the growth (or lack thereof) of app use and how much revenue is gained from that to determine how much revenues the app will make in the next year. This can help determine the financial viability of an app and whether or not a change should be made to the app or revenue source.
Predictive models are set up in a series of steps. First relevant predictors are assessed and decided upon, then from these relative predictors data is collected, this data is in turn used to formulate a statistical model. This model is revised and updated upon new data collected after the predictive model was created. The math that these models will utilize can vary greatly depending on the complexity required. Sometimes simple linear equations are all that is needed but in more complex models entire neural networks can be employed and utilized.
The question many business may have is whether or not they can trust the statistics. Whether what this statistical model says about potential revenue effects of actions is accurate and factual. Should, in other words, the numbers be trusted. While it’s true that, to a certain extent, the past can’t always be used to understand the future, it is important to note that these models have been tried and tested in a vast array of fields, from email carriers, to health care providers, to hospitals, to technology firms and overall these models have shown themselves to be remarkably effective. Using statistics to understand the future, especially in relationship to human behavior and future revenues has been incredibly successful and without a doubt is relied upon by many companies.
Predictive modeling is a widely used tool that helps businesses everywhere justify and inform their decisions with reliable and accurate information