关于线性回归的原理请参考Coursera上斯坦福大学Andrew Ng教授的“机器学习公开课”
下面直接到sparkMlib实现线性回归,spark源码下有两个关于机器学习的包,一个是基于RDD的包Mlib-support the RDD-based API in spark.mllib,另外一个也就是spark目前主推的基于 DataFrame-based API,这个在源码的ml包下;基于DataFrame的API更加友好;下面就基于RDD的API进行分析;
训练集数据如下:
-0.4307829,-1.63735562648104 -2.00621178480549 -1.86242597251066 -1.02470580167082 -0.522940888712441 -0.863171185425945 -1.04215728919298 -0.864466507337306 -0.1625189,-1.98898046126935 -0.722008756122123 -0.787896192088153 -1.02470580167082 -0.522940888712441 -0.863171185425945 -1.04215728919298 -0.864466507337306 -0.1625189,-1.57881887548545 -2.1887840293994 1.36116336875686 -1.02470580167082 -0.522940888712441 -0.863171185425945 0.342627053981254 -0.155348103855541 -0.1625189,-2.16691708463163 -0.807993896938655 -0.787896192088153 -1.02470580167082 -0.522940888712441 -0.863171185425945 -1.04215728919298 -0.864466507337306
其中第一个字段(逗号前)的为因变量Y,逗号后为自变量X1,X2,X3等,为了测试方便训练集也当做测试集使用;
public static void main(String[] args) { SparkConf conf =new SparkConf().setAppName("lineRegression").setMaster("local[1]"); JavaSparkContext sc = new JavaSparkContext(conf); String path = "E:/sparkMlib/sparkMlib/src/mllib/ridge-data/lpsa.data"; JavaRDD<String>data = sc.textFile(path); JavaRDD<LabeledPoint> parseData = data.map(new Function<String,LabeledPoint>(){ public LabeledPoint call(String line) throws Exception { String [] part = line.split(","); //设置特征 String[] features = part[1].split(" "); double [] v =new double[features.length-1]; for(int i=0;i<features.length-1;i++){ v[i]=Double.parseDouble(features[i]); } return new LabeledPoint(Double.parseDouble(part[0]),(Vector) Vectors.dense(v)); } }); parseData.cache(); // 建立模型 int numIterations = 500; double stepSize = 0.00000001; final org.apache.spark.mllib.regression.LinearRegressionModel model = LinearRegressionWithSGD.train(JavaRDD.toRDD(parseData), numIterations,stepSize); JavaRDD<Tuple2<Double, Double>> valuesAndPreds = parseData.map( new Function<LabeledPoint, Tuple2<Double, Double>>() { public Tuple2<Double, Double> call(LabeledPoint point) { double prediction = model.predict(point.features()); //打印预测值和实际值 System.out.println(prediction+":"+point.label()); return new Tuple2<Double, Double>(prediction, point.label()); } } ); //获取均方误差 double MSE = new JavaDoubleRDD(valuesAndPreds.map( new Function<Tuple2<Double, Double>, Object>() { public Object call(Tuple2<Double, Double> pair) { return Math.pow(pair._1() - pair._2(), 2.0); } } ).rdd()).mean(); System.out.println("training Mean Squared Error = " + MSE); }
打印结果:
下面就是调优;
原创文章,作者:Maggie-Hunter,如若转载,请注明出处:https://blog.ytso.com/9383.html