[2024年11月]更新のDatabricks Databricks-Certified-Professional-Data-EngineerテストエンジンとPDFで完全版無料問題集を無料提供
最新版を今すぐ試そうDatabricks-Certified-Professional-Data-Engineer認定有効な試験問題集
DataBricks認定プロフェッショナルデータエンジニア認定は、DataBricksを使用するデータエンジニアにとって貴重な資格です。候補者はDatabrickを深く理解しており、複雑なデータエンジニアリングの問題を解決するために効果的に使用できることを示しています。この認定は、データエンジニアがキャリアを前進させ、収益の可能性を高め、ビッグデータと機械学習の分野の専門家として認識を獲得するのに役立ちます。
質問 # 40
The data governance team is reviewing user for deleting records for compliance with GDPR. The following logic has been implemented to propagate deleted requests from the user_lookup table to the user aggregate table.
Assuming that user_id is a unique identifying key and that all users have requested deletion have been removed from the user_lookup table, which statement describes whether successfully executing the above logic guarantees that the records to be deleted from the user_aggregates table are no longer accessible and why?
- A. Yes: Delta Lake ACID guarantees provide assurance that the DELETE command successed fully and permanently purged these records.
- B. No: the change data feed only tracks inserts and updates not deleted records.
- C. No: files containing deleted records may still be accessible with time travel until a BACUM command is used to remove invalidated data files.
- D. No: the Delta Lake DELETE command only provides ACID guarantees when combined with the MERGE INTO command
正解：C
解説：
The DELETE operation in Delta Lake is ACID compliant, which means that once the operation is successful, the records are logically removed from the table. However, the underlying files that contained these records may still exist and be accessible via time travel to older versions of the table. To ensure that these records are physically removed and compliance with GDPR is maintained, a VACUUM command should be used to clean up these data files after a certain retention period. The VACUUM command will remove the files from the storage layer, and after this, the records will no longer be accessible.
質問 # 41
A junior developer complains that the code in their notebook isn't producing the correct results in the development environment. A shared screenshot reveals that while they're using a notebook versioned with Databricks Repos, they're using a personal branch that contains old logic. The desired branch named dev-2.3.9 is not available from the branch selection dropdown.
Which approach will allow this developer to review the current logic for this notebook?
- A. Merge all changes back to the main branch in the remote Git repository and clone the repo again
- B. Use Repos to make a pull request use the Databricks REST API to update the current branch to dev-2.3.9
- C. Use Repos to checkout the dev-2.3.9 branch and auto-resolve conflicts with the current branch
- D. Use Repos to pull changes from the remote Git repository and select the dev-2.3.9 branch.
- E. Use Repos to merge the current branch and the dev-2.3.9 branch, then make a pull request to sync with the remote repository
正解：D
解説：
This is the correct answer because it will allow the developer to update their local repository with the latest changes from the remote repository and switch to the desired branch. Pulling changes will not affect the current branch or create any conflicts, as it will only fetch the changes and not merge them. Selecting the dev-2.3.9 branch from the dropdown will checkout that branch and display its contents in the notebook. Verified Reference: [Databricks Certified Data Engineer Professional], under "Databricks Tooling" section; Databricks Documentation, under "Pull changes from a remote repository" section.
質問 # 42
You are asked to debug a databricks job that is taking too long to run on Sunday's, what are the steps you are going to take to identify the step that is taking longer to run?
- A. Once a job is launched, you cannot access the job's notebook activity.
- B. Use the compute's spark UI to monitor the job activity.
- C. Enable debug mode in the Jobs to see the output activity of a job, output should be available to view.
- D. Under Workflow UI and jobs select job you want to monitor and select the run, notebook activity can be viewed.
- E. A notebook activity of job run is only visible when using all-purpose cluster.
正解：D
解説：
Explanation
The answer is, Under Workflow UI and jobs select job you want to monitor and select the run, notebook activity can be viewed.
You have the ability to view current active runs or completed runs, once you click the run you can see the A picture containing graphical user interface Description automatically generated
Click on the run to view the notebook output
Graphical user interface, text, application, email Description automatically generated
質問 # 43
Which of the statement is correct about the cluster pools?
- A. Cluster pools allow you to save time when starting a new cluster
- B. Cluster pools allow you to create a cluster
- C. Cluster pools allow you to perform load balancing
- D. Cluster pools allow you to have all the nodes in the cluster from single physical server rack
- E. Cluster pools are used to share resources among multiple teams
正解：A
質問 # 44
A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records.
In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?
- A. VACUUM the Delta table after each batch completes.
- B. Rely on Delta Lake schema enforcement to prevent duplicate records.
- C. Perform an insert-only merge with a matching condition on a unique key.
- D. Perform a full outer join on a unique key and overwrite existing data.
- E. Set the configuration delta.deduplicate = true.
正解：C
解説：
To deduplicate data against previously processed records as it is inserted into a Delta table, you can use the merge operation with an insert-only clause. This allows you to insert new records that do not match any existing records based on a unique key, while ignoring duplicate records that match existing records. For example, you can use the following syntax:
MERGE INTO target_table USING source_table ON target_table.unique_key = source_table.unique_key WHEN NOT MATCHED THEN INSERT * This will insert only the records from the source table that have a unique key that is not present in the target table, and skip the records that have a matching key. This way, you can avoid inserting duplicate records into the Delta table.
Reference:
https://docs.databricks.com/delta/delta-update.html#upsert-into-a-table-using-merge
https://docs.databricks.com/delta/delta-update.html#insert-only-merge
質問 # 45
Which of the following programming languages can be used to build a Databricks SQL dashboard?
- A. R
- B. SQL
- C. Python
- D. All of the above
- E. Scala
正解：B
質問 # 46
What is the main difference between the silver layer and gold layer in medallion architecture?
- A. Silver may contain aggregated data, gold may preserve the granularity of original data
- B. Silver optimized to perform ETL, Gold is optimized query performance
- C. Silver is stored in Delta Lake, Gold is stored in memory
- D. Gold is optimized go perform ETL, Silver is optimized for query performance
- E. Silver is copy of Bronze, Gold is a copy of Silver
正解：B
解説：
Explanation
Medallion Architecture - Databricks
Gold Layer:
1. Powers Ml applications, reporting, dashboards, ad hoc analytics
2. Refined views of data, typically with aggregations
3. Reduces strain on production systems
4. Optimizes query performance for business-critical data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
A diagram of a house Description automatically generated with low confidence
質問 # 47
A Delta Lake table was created with the below query:
Consider the following query:
DROP TABLE prod.sales_by_store -
If this statement is executed by a workspace admin, which result will occur?
- A. The table will be removed from the catalog and the data will be deleted.
- B. Nothing will occur until a COMMIT command is executed.
- C. The table will be removed from the catalog but the data will remain in storage.
- D. An error will occur because Delta Lake prevents the deletion of production data.
- E. Data will be marked as deleted but still recoverable with Time Travel.
正解：A
解説：
When a table is dropped in Delta Lake, the table is removed from the catalog and the data is deleted. This is because Delta Lake is a transactional storage layer that provides ACID guarantees. When a table is dropped, the transaction log is updated to reflect the deletion of the table and the data is deleted from the underlying storage. References:
* https://docs.databricks.com/delta/quick-start.html#drop-a-table
* https://docs.databricks.com/delta/delta-batch.html#drop-table
質問 # 48
A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFramedf. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Events are recorded once per minute per device.
Streaming DataFramedfhas the following schema:
"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT"
Code block:
Choose the response that correctly fills in the blank within the code block to complete this task.
- A. "event_time"
- B. window("event_time", "10 minutes").alias("time")
- C. lag("event_time", "10 minutes").alias("time")
- D. window("event_time", "5 minutes").alias("time")
- E. to_interval("event_time", "5 minutes").alias("time")
正解：D
解説：
This is the correct answer because the window function is used to group streaming data by time intervals. The window function takes two arguments: a time column and a window duration. The window duration specifies how long each window is, and must be a multiple of 1 second. In this case, the window duration is "5 minutes", which means each window will cover a non-overlapping five-minute interval. The window function also returns a struct column with two fields: start and end, which represent the start and end time of each window. The alias function is used to rename the struct column as "time". Verified References: [Databricks Certified Data Engineer Professional], under "Structured Streaming" section; Databricks Documentation, under "WINDOW" section.https://www.databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-struc
質問 # 49
Which of the following Structured Streaming queries successfully performs a hop from a Silver to Gold table?
- A. 1.(spark.read.load(rawSalesLocation)
2. .writeStream
3. .option("checkpointLocation", checkpointPath)
4. .outputMode("append")
5. .table("uncleanedSales") ) - B. 1.(spark.table("sales")
2..writeStream
3..option("checkpointLocation", checkpointPath)
4..outputMode("complete")
5..table("sales") ) - C. 1.(spark.readStream.load(rawSalesLocation)
2..writeStream
3..option("checkpointLocation", checkpointPath)
4..outputMode("append")
5..table("uncleanedSales") ) - D. 1.(spark.table("sales")
2..groupBy("store")
3..agg(sum("sales"))
4..writeStream
5..option("checkpointLocation", checkpointPath)
6..outputMode("complete")
7..table("aggregatedSales") )
(Correct) - E. 1.(spark.table("sales")
2..withColumn("avgPrice", col("sales") / col("units"))
3..writeStream
4..option("checkpointLocation", checkpointPath)
5..outputMode("append")
6..table("cleanedSales") )
正解：D
解説：
Explanation
The answer is
1.(spark.table("sales")
2..groupBy("store")
3..agg(sum("sales"))
4..writeStream
5..option("checkpointLocation", checkpointPath)
6..outputMode("complete")
7..table("aggregatedSales") )
The gold layer is normally used to store aggregated data
Review the below link for more info,
Medallion Architecture - Databricks
Gold Layer:
1. Powers Ml applications, reporting, dashboards, ad hoc analytics
2. Refined views of data, typically with aggregations
3. Reduces strain on production systems
4. Optimizes query performance for business-critical data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
A diagram of a house Description automatically generated with low confidence
質問 # 50
Which of the following features of data lakehouse can help you meet the needs of both workloads?
- A. Data lakehouse provides autoscaling for compute clusters.
- B. Data lakehouse fully exists in the cloud.
- C. Data lakehouse requires very little data modeling.
- D. Data lakehouse combines compute and storage for simple governance.
- E. Data lakehouse can store unstructured data and support ACID transactions.
正解：E
解説：
Explanation
The answer is A data lakehouse stores unstructured data and is ACID-compliant,
質問 # 51
Which of the following data workloads will utilize a Bronze table as its destination?
- A. A job that enriches data by parsing its timestamps into a human-readable format
- B. A job that develops a feature set for a machine learning application
- C. A job that queries aggregated data to publish key insights into a dashboard
- D. A job that ingests raw data from a streaming source into the Lakehouse
- E. A job that aggregates cleaned data to create standard summary statistics
正解：D
解説：
Explanation
The answer is A job that ingests raw data from a streaming source into the Lakehouse.
The ingested data from the raw streaming data source like Kafka is first stored in the Bronze layer as first destination before it is further optimized and stored in Silver.
Medallion Architecture - Databricks
Bronze Layer:
1. Raw copy of ingested data
2. Replaces traditional data lake
3. Provides efficient storage and querying of full, unprocessed history of data
4. No schema is applied at this layer
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
Purpose of each layer in medallion architecture
質問 # 52
You have written a notebook to generate a summary data set for reporting, Notebook was scheduled using the job cluster, but you realized it takes 8 minutes to start the cluster, what feature can be used to start the cluster in a timely fashion so your job can run immediatley?
- A. Disable auto termination so the cluster is always running
- B. Pin the cluster in the cluster UI page so it is always available to the jobs
- C. Use the Databricks cluster pools feature to reduce the startup time
- D. Setup an additional job to run ahead of the actual job so the cluster is running second job starts
- E. Use Databricks Premium edition instead of Databricks standard edition
正解：C
解説：
Explanation
Cluster pools allow us to reserve VM's ahead of time, when a new job cluster is created VM are grabbed from the pool. Note: when the VM's are waiting to be used by the cluster only cost incurred is Azure. Databricks run time cost is only billed once VM is allocated to a cluster.
Here is a demo of how to setup a pool and follow some best practices,
Graphical user interface, text Description automatically generated
質問 # 53
You are currently working on a notebook that will populate a reporting table for downstream process consumption, this process needs to run on a schedule every hour, what type of cluster are you going to use to set up this job?
- A. Use delta live table pipeline to run in continuous mode
- B. The job cluster is best suited for this purpose.
- C. Since it's just a single job and we need to run every hour, we can use an all-purpose cluster
- D. Use Azure VM to read and write delta tables in Python
正解：B
解説：
Explanation
The answer is, The Job cluster is best suited for this purpose.
Since you don't need to interact with the notebook during the execution especially when it's a scheduled job, job cluster makes sense. Using an all-purpose cluster can be twice as expensive as a job cluster.
FYI,
When you run a job scheduler with option of creating a new cluster when the job is complete it terminates the cluster. You cannot restart a job cluster.
質問 # 54
The data governance team has instituted a requirement that all tables containing Personal Identifiable Information (PH) must be clearly annotated. This includes adding column comments, table comments, and setting the custom table property"contains_pii" = true.
The following SQL DDL statement is executed to create a new table:
Which command allows manual confirmation that these three requirements have been met?
- A. SHOW TABLES dev
- B. SHOW TBLPROPERTIES dev.pii test
- C. DESCRIBE HISTORY dev.pii test
- D. DESCRIBE EXTENDED dev.pii test
- E. DESCRIBE DETAIL dev.pii test
正解：D
解説：
Explanation
This is the correct answer because it allows manual confirmation that these three requirements have been met.
The requirements are that all tables containing Personal Identifiable Information (PII) must be clearly annotated, which includes adding column comments, table comments, and setting the custom table property
"contains_pii" = true. The DESCRIBE EXTENDED command is used to display detailed information about a table, such as its schema, location, properties, and comments. By using this command on the dev.pii_test table, one can verify that the table has been created with the correct column comments, table comment, and custom table property as specified in the SQL DDL statement. Verified References: [Databricks Certified Data Engineer Professional], under "Lakehouse" section; Databricks Documentation, under "DESCRIBE EXTENDED" section.
質問 # 55
The Delta Live Tables Pipeline is configured to run in Development mode using the Triggered Pipeline Mode.
what is the expected outcome after clicking Start to update the pipeline?
- A. All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will persist after the pipeline is stopped to allow for additional development and testing
- B. All datasets will be updated once and the pipeline will shut down. The compute resources will persist to allow for additional development and testing
- C. All datasets will be updated continuously and the pipeline will not shut down. The compute resources will persist with the pipeline
- D. All datasets will be updated at set intervals until the pipeline is shut down. The compute resources will be deployed for the update and terminated when the pipeline is stopped
- E. All datasets will be updated once and the pipeline will shut down. The compute resources will be terminated
正解：C
解説：
Explanation
The answer is All datasets will be updated once and the pipeline will shut down. The compute re-sources will persist to allow for additional testing.
DLT pipeline supports two modes Development and Production, you can switch between the two based on the stage of your development and deployment lifecycle.
Development and production modes
When you run your pipeline in development mode, the Delta Live Tables system:
*Reuses a cluster to avoid the overhead of restarts.
*Disables pipeline retries so you can immediately detect and fix errors.
In production mode, the Delta Live Tables system:
*Restarts the cluster for specific recoverable errors, including memory leaks and stale credentials.
*Retries execution in the event of specific errors, for example, a failure to start a cluster.
Use the buttons in the Pipelines UI to switch between develop-ment and production modes. By default, pipelines run in development mode.
Switching between development and production modes only controls cluster and pipeline execution behavior.
Storage locations must be configured as part of pipeline settings and are not affected when switching between modes.
Please review additional DLT concepts using below link
https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-concepts.html#delta-live-tables-c
質問 # 56
The data engineer team has been tasked with configured connections to an external database that does not have a supported native connector with Databricks. The external database already has data security configured by group membership. These groups map directly to user group already created in Databricks that represent various teams within the company.
A new login credential has been created for each group in the external database. The Databricks Utilities Secrets module will be used to make these credentials available to Databricks users.
Assuming that all the credentials are configured correctly on the external database and group membership is properly configured on Databricks, which statement describes how teams can be granted the minimum necessary access to using these credentials?
- A. "Read" permissions should be set on a secret scope containing only those credentials that will be used by a given team.
- B. No additional configuration is necessary as long as all users are configured as administrators in the workspace where secrets have been added.
- C. "Manage" permission should be set on a secret scope containing only those credentials that will be used by a given team.
- D. ''Read'' permissions should be set on a secret key mapped to those credentials that will be used by a given team.
正解：A
解説：
In Databricks, using the Secrets module allows for secure management of sensitive information such as database credentials. Granting 'Read' permissions on a secret key that maps to database credentials for a specific team ensures that only members of that team can access these credentials. This approach aligns with the principle of least privilege, granting users the minimum level of access required to perform their jobs, thus enhancing security.
Reference:
Databricks Documentation on Secret Management: Secrets
質問 # 57
Which statement regarding spark configuration on the Databricks platform is true?
- A. Spark configuration properties set for an interactive cluster with the Clusters UI will impact all notebooks attached to that cluster.
- B. Spark configuration set within an notebook will affect all SparkSession attached to the same interactive cluster
- C. The Databricks REST API can be used to modify the Spark configuration properties for an interactive cluster without interrupting jobs.
- D. When the same spar configuration property is set for an interactive to the same interactive cluster.
正解：A
解説：
When Spark configuration properties are set for an interactive cluster using the Clusters UI in Databricks, those configurations are applied at the cluster level. This means that all notebooks attached to that cluster will inherit and be affected by these configurations. This approach ensures consistency across all executions within that cluster, as the Spark configuration properties dictate aspects such as memory allocation, number of executors, and other vital execution parameters. This centralized configuration management helps maintain standardized execution environments across different notebooks, aiding in debugging and performance optimization.
Reference:
Databricks documentation on configuring clusters: https://docs.databricks.com/clusters/configure.html
質問 # 58
You had AUTO LOADER to process millions of files a day and noticed slowness in load process, so you scaled up the Databricks cluster but realized the performance of the Auto loader is still not improving, what is the best way to resolve this.
- A. Setup a second AUTO LOADER process to process the data
- B. Merge files to one large file
- C. Increase the maxFilesPerTrigger option to a sufficiently high number
- D. AUTO LOADER is not suitable to process millions of files a day
- E. Copy the data from cloud storage to local disk on the cluster for faster access
正解：C
解説：
Explanation
The default value of maxFilesPerTrigger is 1000 it can be increased to a much higher number but will require a much larger compute to process.
Graphical user interface, text, application, email Description automatically generated
https://docs.databricks.com/ingestion/auto-loader/options.html
質問 # 59
Which of the following SQL statements can be used to update a transactions table, to set a flag on the table from Y to N
- A. UPDATE transactions SET active_flag = 'N' WHERE active_flag = 'Y'
- B. MERGE transactions SET active_flag = 'N' WHERE active_flag = 'Y'
- C. MODIFY transactions SET active_flag = 'N' WHERE active_flag = 'Y'
- D. REPLACE transactions SET active_flag = 'N' WHERE active_flag = 'Y'
正解：D
解説：
Explanation
The answer is
UPDATE transactions SET active_flag = 'N' WHERE active_flag = 'Y'
Delta Lake supports UPDATE statements on the delta table, all of the changes as part of the update are ACID compliant.
質問 # 60
Which of the statements are incorrect when choosing between lakehouse and Datawarehouse?
- A. Lakehouse can be accessed through various API's including but not limited to Py-thon/R/SQL
- B. Lakehouse cannot serve low query latency with high reliability for BI workloads, only suitable for batch workloads.
- C. Traditional Data warehouses have storage and compute are coupled.
- D. Lakehouse uses standard data formats like Parquet.
- E. Lakehouse can have special indexes and caching which are optimized for Machine learning
正解：B
解説：
Explanation
The answer is Lakehouse cannot serve low query latency with high reliability for BI workloads, only suitable for batch workloads.
Lakehouse can replace traditional warehouses by leveraging storage and compute optimizations like caching to serve them with low query latency with high reliability.
Focus on comparisons between Spark Cache vs Delta Cache.
https://docs.databricks.com/delta/optimizations/delta-cache.html
What Is a Lakehouse? - The Databricks Blog
Graphical user interface, text, application Description automatically generated
Bottom of Form
Top of Form
質問 # 61
......
100%合格保証付きの素晴らしいDatabricks-Certified-Professional-Data-Engineer試験問題PDF：https://www.passtest.jp/Databricks/Databricks-Certified-Professional-Data-Engineer-shiken.html
Databricks-Certified-Professional-Data-Engineer問題集で2024年最新のDatabricks試験問題：https://drive.google.com/open?id=17yNT7ja8WeEEAJInqtIquu9DNikk75fZ