Hudi 0.14.0編譯

1 編譯環(huán)境

2 hudi準(zhǔn)備

2.1 源碼

$ git clone https://github.com/apache/hudi.git
$ cd hudi
$ git checkout release-0.14.0

2.2 修改pom文件

2.2.1 新增repository加速依賴下載

<repository>
  <id>nexus-aliyun</id>
  <name>nexus-aliyun</name>
  <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
  <releases>
    <enabled>true</enabled>
  </releases>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
</repository>

2.2.2  修改依賴組件版本

hudi/pom.xml

更改spark版本為3.2.3，flink版本為1.16.0

<spark32.version>3.2.3</spark32.version>
<flink1.16.version>1.16.0</flink1.16.version>

hudi/packaging/hudi-flink-bundle/pom.xml

更改hive版本為3.1.3

參考鏈接：Hive Metastore | Apache Hudi

<profile>
  <id>flink-bundle-shade-hive3</id>
  <properties>
    <hive.version>3.1.3</hive.version>
    <flink.bundle.hive.scope>compile</flink.bundle.hive.scope>
  </properties>
  <dependencies>
    <dependency>
      <groupId>${hive.groupid}</groupId>
      <artifactId>hive-service-rpc</artifactId>
      <version>${hive.version}</version>
      <scope>${flink.bundle.hive.scope}</scope>
    </dependency>
    <dependency>
      <groupId>${hive.groupid}</groupId>
      <artifactId>hive-standalone-metastore</artifactId>
      <version>${hive.version}</version>
      <scope>${flink.bundle.hive.scope}</scope>
    </dependency>
  </dependencies>
</profile>

3 編譯

3.1 編譯命令

mvn clean package \
-DskipTests \
-Dcheckstyle.skip \
-Dhadoop.version=3.3.4 \
-Dhive.version=3.1.3 \
-Dscala-2.12 \
-Dspark3.2 \
-Dflink1.16 \
-Pflink-bundle-shade-hive3

3.2 編譯成功

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hudi 0.14.0:
[INFO] 
[INFO] Hudi ............................................... SUCCESS [ 44.589 s]
[INFO] hudi-tests-common .................................. SUCCESS [01:07 min]
[INFO] hudi-common ........................................ SUCCESS [01:25 min]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 18.216 s]
[INFO] hudi-sync-common ................................... SUCCESS [  5.951 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [ 26.903 s]
[INFO] hudi-aws ........................................... SUCCESS [ 12.256 s]
[INFO] hudi-timeline-service .............................. SUCCESS [  5.442 s]
[INFO] hudi-client ........................................ SUCCESS [  0.056 s]
[INFO] hudi-client-common ................................. SUCCESS [ 19.466 s]
[INFO] hudi-spark-client .................................. SUCCESS [01:04 min]
[INFO] hudi-spark-datasource .............................. SUCCESS [  0.028 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [01:32 min]
[INFO] hudi-spark3-common ................................. SUCCESS [ 25.230 s]
[INFO] hudi-spark3.2plus-common ........................... SUCCESS [ 21.612 s]
[INFO] hudi-spark3.2.x_2.12 ............................... SUCCESS [ 36.385 s]
[INFO] hudi-java-client ................................... SUCCESS [  8.972 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [01:25 min]
[INFO] hudi-gcp ........................................... SUCCESS [  7.062 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [ 32.803 s]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [01:19 min]
[INFO] hudi-cli ........................................... SUCCESS [ 27.285 s]
[INFO] hudi-flink-client .................................. SUCCESS [ 19.600 s]
[INFO] hudi-datahub-sync .................................. SUCCESS [  3.451 s]
[INFO] hudi-adb-sync ...................................... SUCCESS [  6.029 s]
[INFO] hudi-sync .......................................... SUCCESS [  0.021 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [ 35.741 s]
[INFO] hudi-datahub-sync-bundle ........................... SUCCESS [ 40.215 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [ 32.659 s]
[INFO] hudi-aws-bundle .................................... SUCCESS [ 48.612 s]
[INFO] hudi-gcp-bundle .................................... SUCCESS [ 32.596 s]
[INFO] hudi-spark3.2-bundle_2.12 .......................... SUCCESS [01:00 min]
[INFO] hudi-presto-bundle ................................. SUCCESS [ 41.722 s]
[INFO] hudi-utilities-slim-bundle_2.12 .................... SUCCESS [01:01 min]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [ 47.789 s]
[INFO] hudi-trino-bundle .................................. SUCCESS [ 37.650 s]
[INFO] hudi-examples ...................................... SUCCESS [  0.023 s]
[INFO] hudi-examples-common ............................... SUCCESS [  6.493 s]
[INFO] hudi-examples-spark ................................ SUCCESS [ 17.247 s]
[INFO] hudi-flink-datasource .............................. SUCCESS [  0.028 s]
[INFO] hudi-flink1.16.x ................................... SUCCESS [  4.574 s]
[INFO] hudi-flink ......................................... SUCCESS [ 14.562 s]
[INFO] hudi-examples-flink ................................ SUCCESS [  7.564 s]
[INFO] hudi-examples-java ................................. SUCCESS [  9.664 s]
[INFO] hudi-flink1.13.x ................................... SUCCESS [  6.623 s]
[INFO] hudi-flink1.14.x ................................... SUCCESS [  5.684 s]
[INFO] hudi-flink1.15.x ................................... SUCCESS [  4.878 s]
[INFO] hudi-flink1.17.x ................................... SUCCESS [  5.485 s]
[INFO] hudi-kafka-connect ................................. SUCCESS [  8.019 s]
[INFO] hudi-flink1.16-bundle .............................. SUCCESS [01:54 min]
[INFO] hudi-kafka-connect-bundle .......................... SUCCESS [01:20 min]
[INFO] hudi-cli-bundle_2.12 ............................... SUCCESS [ 44.297 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  26:05 min
[INFO] Finished at: 2023-10-24T14:54:46+08:00
[INFO] ------------------------------------------------------------------------

編譯成功后，進(jìn)入hudi-cli說明成功。

[root@localhost hudi]# ./hudi-cli/hudi-cli.sh

編譯完成后，相關(guān)的包在packaging目錄的各個(gè)模塊中：

[root@localhost packaging]# pwd
/root/app/hudi/packaging
[root@localhost packaging]# ll
總用量 4
drwxr-xr-x. 11 root root  247 10月 24 11:11 bundle-validation
drwxr-xr-x.  4 root root   46 10月 24 14:43 hudi-aws-bundle
drwxr-xr-x.  5 root root   89 10月 24 14:54 hudi-cli-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:42 hudi-datahub-sync-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:50 hudi-flink-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:44 hudi-gcp-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:42 hudi-hadoop-mr-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:43 hudi-hive-sync-bundle
drwxr-xr-x.  3 root root   32 10月 24 13:45 hudi-integ-test-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:52 hudi-kafka-connect-bundle
drwxr-xr-x.  2 root root   21 10月 24 13:45 hudi-metaserver-server-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:46 hudi-presto-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:45 hudi-spark-bundle
drwxr-xr-x.  4 root root  101 10月 24 14:48 hudi-timeline-server-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:48 hudi-trino-bundle
drwxr-xr-x.  4 root root   46 10月 24 14:39 hudi-utilities-bundle
drwxr-xr-x.  4 root root   63 10月 24 14:46 hudi-utilities-slim-bundle
-rw-r--r--.  1 root root 2206 10月 24 11:11 README.md

比如，flink與hudi的包，hudi-flink1.16-bundle-0.14.0.jar：

[root@localhost target]# pwd
/root/app/hudi/packaging/hudi-flink-bundle/target
[root@localhost target]# ll
總用量 135228
drwxr-xr-x. 4 root root       33 10月 24 14:50 classes
-rw-r--r--. 1 root root    13474 10月 24 14:52 dependency-reduced-pom.xml
drwxr-xr-x. 3 root root       25 10月 24 14:50 generated-sources
-rw-r--r--. 1 root root 96299310 10月 24 14:52 hudi-flink1.16-bundle-0.14.0.jar
-rw-r--r--. 1 root root 42111892 10月 24 14:52 hudi-flink1.16-bundle-0.14.0-sources.jar
drwxr-xr-x. 2 root root       28 10月 24 14:50 maven-archiver
drwxr-xr-x. 3 root root       22 10月 24 14:50 maven-shared-archive-resources
drwxr-xr-x. 3 root root       35 10月 24 14:50 maven-status
-rw-r--r--. 1 root root    18561 10月 24 14:50 original-hudi-flink1.16-bundle-0.14.0.jar
-rw-r--r--. 1 root root    14046 10月 24 14:50 original-hudi-flink1.16-bundle-0.14.0-sources.jar
-rw-r--r--. 1 root root      809 10月 24 14:50 rat.txt
drwxr-xr-x. 3 root root       22 10月 24 14:50 test-classes