Cloudera CDP Data Engineer - Certification - CDP-3002 模擬練習

A. Pre-partition tables based on the join columns for faster data co-location

B. Use broadcast joins for small tables involved in the join operation

C. All of the above

D. Increase the number of Spark executors without any further optimization

A. Implement custom logic within each task to send notifications using external libraries.

B. Use the EmailOperator to send email notifications with details about the task status.

C. Configure the Airflow web Ul to send alerts based on task execution status.

D. Utilize Airflow variables to store notification details and access them within tasks.

A. Both A and B.

B. By directly writing SQL queries against distributed data in Spark.

C. By implementing custom logic in RDDs for data manipulation.

D. By using Spark ML libraries for machine learning tasks on data.

A. Manual intervention and rerun via the Airflow Webserver

B. The on failure callback function in DAG definitions

C. Airflow Scheduler's automatic rerun feature

D. The retry_delay and retries parameters in task definitions

A. Optimistic locking in Iceberg introduces a significant performance overhead.

B. Optimistic locking might result in many failed transactions and retries, potentially leading to job instability.

C. Iceberg's optimistic locking requires manual conflict resolution.

D. Iceberg's optimistic locking isn't suitable for distributed environments like CDP.

A. Use the FailoverOperator to automatically skip downstream tasks if the validation check fails.

B. Leverage XCom to share the validation results with downstream tasks for further processing.

C. Configure the DAG to send email notifications upon validation failures for manual intervention.

D. Implement the validation checks within custom Python operators and raise exceptions upon failure, triggering downstream tasks to handle the error.

A. A programming language used for creating data pipelines

B. A type of operator used for executing Python functions

C. A visualization tool for monitoring pipeline execution

D. A collection of tasks with directional dependencies

A. Manual scripting

B. Apache NiFi

C. Apache Avro

D. Traditional RDBMS

E. Apache Spark

A. Enable Spark broadcast joins

B. Increase driver memory (spark.driver.memory)

C. Increase executor memory (spark.executor.memory)

D. Upgrade to a larger Kubernetes pod size.

E. Decrease the size of Spark partitions (spark.sql.shuffle.partitions)

A. The DataFrames have the same column names but different data types

B. The DataFrames have different column names but the same data types

C. All of the above

D. The join condition references non-existent columns in one of the DataFrames

A. Leverage partition pruning to only read relevant data from the table

B. Implement custom data compression logic within Spark for improved storage efficiency

C. Increase the number of Spark executors without further optimization

D. Use a different file format for the Hive table, like CSV, for faster parsing

A. To enable communication between different tasks within the same DAG.

B. To execute commands across different operating systems.

C. To trigger external systems when a DAG reaches a specific state.

D. To manage dependencies and package installations across tasks.

A. Monitor the resource utilization of the entire system, assuming it reflects the pipeline's performance.

B. Implement custom monitoring scripts to track specific metrics not readily available in Airflow.

C. You cannot directly monitor Airflow pipelines; rely on external monitoring tools for the underlying infrastructure.

D. Utilize Airflow's built-in metrics and monitoring features like the Airflow web UI and command-line tools.