Spark学习:为Sparksql添加自定义命令

• • 一、修改SqlBase4添加语法规则
  • 二、 Maven编译antlr
  • 三、修改SqlParsqlParser.scala中
  • 四、运行

#修改四处
statement
| SHOW VERSION                                                     #showVersion

ansiNonReserved
| VERSION

nonReserved
| VERSION

//--SPARK-KEYWORD-LIST-START
VERSION: 'VERSION' | 'V';

1、增加visitShowVersion方法

override def visitShowVersion(ctx: ShowVersionContext): LogicalPlan = withOrigin(ctx) {
    ShowVersionCommand()
}

2、实现ShowVersionCommand类

package org.apache.spark.sql.execution.command

import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
import org.apache.spark.sql.types.StringType


case class ShowVersionCommand() extends LeafRunnableCommand {

  override val output: Seq[Attribute] =
    Seq(AttributeReference("version", StringType)())

  override def run(sparkSession: SparkSession): Seq[Row] = {
    val sparkVersion = sparkSession.version
    val javaVersion = System.getProperty("java.version")
    val scalaVersion = scala.util.Properties.releaseVersion
    val output = "Spark Version: %s, Java Version: %s, Scala Version: %s"
      .format(sparkVersion, javaVersion, scalaVersion.getOrElse(""))
    Seq(Row(output))
  }
}

1、powershell中运行spark-sql报错

Failed to load class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver

网上查了查后,是缺少与hive相关的依赖包

//源码根目录重新编译后会额外多生成两个jar包
mvn clean package -DskipTests -Phive -Phive-thriftserver

2、结果
然后重新编译了一遍

mvn package -DskipTests -Phive -Phive-thriftserver