2023年最新の認定サンプル問題Databricks-Certified-Professional-Data-Engineer問題集と練習試験合格させます
Databricks-Certified-Professional-Data-Engineer豪華セット学習ガイドにはオンライン試験エンジン
Databricks Certified Professional Data Engineerは、Databricksを使用してビッグデータを操作する知識とスキルを測定する認定試験です。Databricksは、データエンジニアがビッグデータ処理パイプラインを構築、展開、および管理することができるクラウドベースのデータ処理プラットフォームです。認定試験は、Databricksで作業するデータエンジニアの専門知識を検証するために設計されています。
質問 # 21
Which of the following locations hosts the driver and worker nodes of a Databricks-managed clus-ter?
- A. JDBC data source
- B. Databricks web application
- C. Control plane
- D. Data plane
- E. Databricks Filesystem
正解：D
解説：
Explanation
See the Databricks high-level architecture
質問 # 22
Identify one of the below statements that can query a delta table in PySpark Dataframe API
- A. Spark.read.format("delta").TableAs("table_name")
- B. Spark.read.table.delta("table_name")
- C. Spark.read.mode("delta").table("table_name")
- D. Spark.read.format("delta").LoadTableAs("table_name")
- E. Spark.read.table("table_name")
正解：E
質問 # 23
A new data engineer [email protected] has been assigned to an ELT project. The new data
engineer will need full privileges on the table sales to fully manage the project.
Which of the following commands can be used to grant full permissions on the table to the new data engineer?
- A. 1. GRANT SELECT CREATE MODIFY ON TABLE sales TO [email protected];
- B. 1. GRANT ALL PRIVILEGES ON TABLE sales TO [email protected];
- C. 1. GRANT SELECT ON TABLE sales TO [email protected];
- D. 1. GRANT ALL PRIVILEGES ON TABLE [email protected] TO sales;
- E. 1. GRANT USAGE ON TABLE sales TO [email protected];
正解：B
質問 # 24
You would like to build a spark streaming process to read from a Kafka queue and write to a Delta table every
15 minutes, what is the correct trigger option
- A. trigger(processingTime = "15 Minutes")
- B. trigger(process "15 minutes")
- C. trigger(15)
- D. trigger("15 minutes")
- E. trigger(processingTime = 15)
正解：A
解説：
Explanation
The answer is trigger(processingTime = "15 Minutes")
Triggers:
*Unspecified
This is the default. This is equivalent to using processingTime="500ms"
*Fixed interval micro-batches .trigger(processingTime="2 minutes")
The query will be executed in micro-batches and kicked off at the user-specified intervals
*One-time micro-batch .trigger(once=True)
The query will execute a single micro-batch to process all the available data and then stop on its own
*One-time micro-batch.trigger .trigger(availableNow=True) -- New feature a better version of (once=True) Databricks supports trigger(availableNow=True) in Databricks Runtime 10.2 and above for Delta Lake and Auto Loader sources. This functionality combines the batch processing approach of trigger once with the ability to configure batch size, resulting in multiple parallelized batches that give greater control for right-sizing batches and the resultant files.
質問 # 25
A data engineer has ingested data from an external source into a PySpark DataFrame raw_df. They need to
briefly make this data available in SQL for a data analyst to perform a quality assurance check on the data.
Which of the following commands should the data engineer run to make this data available in SQL for only
the remainder of the Spark session?
- A. raw_df.createTable("raw_df")
- B. raw_df.write.save("raw_df")
- C. There is no way to share data between PySpark and SQL
- D. raw_df.saveAsTable("raw_df")
- E. raw_df.createOrReplaceTempView("raw_df")
正解：E
質問 # 26
Which of the following SQL statements can be used to update a transactions table, to set a flag on the table from Y to N
- A. REPLACE transactions SET active_flag = 'N' WHERE active_flag = 'Y'
- B. MODIFY transactions SET active_flag = 'N' WHERE active_flag = 'Y'
- C. MERGE transactions SET active_flag = 'N' WHERE active_flag = 'Y'
- D. UPDATE transactions SET active_flag = 'N' WHERE active_flag = 'Y'
正解：A
解説：
Explanation
The answer is
UPDATE transactions SET active_flag = 'N' WHERE active_flag = 'Y'
Delta Lake supports UPDATE statements on the delta table, all of the changes as part of the update are ACID compliant.
質問 # 27
Which of the statements are incorrect when choosing between lakehouse and Datawarehouse?
- A. Lakehouse uses standard data formats like Parquet.
- B. Lakehouse can be accessed through various API's including but not limited to Py-thon/R/SQL
- C. Traditional Data warehouses have storage and compute are coupled.
- D. Lakehouse cannot serve low query latency with high reliability for BI workloads, only suitable for batch workloads.
- E. Lakehouse can have special indexes and caching which are optimized for Machine learning
正解：D
解説：
Explanation
The answer is Lakehouse cannot serve low query latency with high reliability for BI workloads, only suitable for batch workloads.
Lakehouse can replace traditional warehouses by leveraging storage and compute optimizations like caching to serve them with low query latency with high reliability.
Focus on comparisons between Spark Cache vs Delta Cache.
https://docs.databricks.com/delta/optimizations/delta-cache.html
What Is a Lakehouse? - The Databricks Blog
Graphical user interface, text, application Description automatically generated
Bottom of Form
Top of Form
質問 # 28
What are the different ways you can schedule a job in Databricks workspace?
- A. Once, Continuous
- B. On-Demand runs, File notification from Cloud object storage
- C. Cron, On Demand runs
- D. Continuous, Incremental
- E. Cron, File notification from Cloud object storage
正解：C
解説：
Explanation
The answer is, Cron, On-Demand runs
Supports running job immediately or using can be scheduled using CRON syntax
質問 # 29
What is the main difference between the bronze layer and silver layer in a medallion architecture?
- A. Silver may contain aggregated data
- B. Duplicates are removed in bronze, schema is applied in silver
- C. Bronze is raw copy of ingested data, silver contains data with production schema and optimized for ELT/ETL throughput
- D. Bad data is filtered in Bronze, silver is a copy of bronze data
正解：C
解説：
Explanation
Medallion Architecture - Databricks
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
A diagram of a house Description automatically generated with low confidence
質問 # 30
A Delta table of weather records is partitioned by date and has the below schema:
date DATE, device_id INT, temp FLOAT, latitude FLOAT, longitude FLOAT
To find all the records from within the Arctic Circle, you execute a query with the below filter:
latitude > 66.3
Which statement describes how the Delta engine identifies which files to load?
- A. The Parquet file footers are scanned for min and max statistics for the latitude column
- B. The Delta log is scanned for min and max statistics for the latitude column
- C. All records are cached to attached storage and then the filter is applied
- D. All records are cached to an operational database and then the filter is applied
- E. The Hive metastore is scanned for min and max statistics for the latitude column
正解：B
解説：
Explanation
This is the correct answer because Delta Lake uses a transaction log to store metadata about each table, including min and max statistics for each column in each data file. The Delta engine can use this information to quickly identify which files to load based on a filter condition, without scanning the entire table or the file footers. This is called data skipping and it can improve query performance significantly. Verified References:
[Databricks Certified Data Engineer Professional], under "Delta Lake" section; [Databricks Documentation], under "Optimizations - Data Skipping" section.
質問 # 31
You have noticed that Databricks SQL queries are running slow, you are asked to look reason why queries are running slow and identify steps to improve the performance, when you looked at the issue you noticed all the queries are running in parallel and using a SQL endpoint(SQL Warehouse) with a single cluster. Which of the following steps can be taken to improve the performance/response times of the queries?
*Please note Databricks recently renamed SQL endpoint to SQL warehouse.
- A. They can turn on the Auto Stop feature for the SQL endpoint(SQL warehouse).
- B. They can turn on the Serverless feature for the SQL endpoint(SQL warehouse) and change the Spot Instance Policy to "Reliability Optimized."
- C. They can increase the maximum bound of the SQL endpoint(SQL warehouse)'s scaling range
- D. They can increase the warehouse size from 2X-Smal to 4XLarge of the SQL end-point(SQL warehouse).
- E. They can turn on the Serverless feature for the SQL endpoint(SQL warehouse).
正解：C
解説：
Explanation
The answer is, They can increase the maximum bound of the SQL endpoint's scaling range when you increase the max scaling range more clusters are added so queries instead of waiting in the queue can start running using available clusters, see below for more explanation.
The question is looking to test your ability to know how to scale a SQL Endpoint(SQL Warehouse) and you have to look for cue words or need to understand if the queries are running sequentially or concurrently. if the queries are running sequentially then scale up(Size of the cluster from 2X-Small to 4X-Large) if the queries are running concurrently or with more users then scale out(add more clusters).
SQL Endpoint(SQL Warehouse) Overview: (Please read all of the below points and the below diagram to understand )
1.A SQL Warehouse should have at least one cluster
2.A cluster comprises one driver node and one or many worker nodes
3.No of worker nodes in a cluster is determined by the size of the cluster (2X -Small ->1 worker, X-Small ->2 workers.... up to 4X-Large -> 128 workers) this is called Scale up
4.A single cluster irrespective of cluster size(2X-Smal.. to ...4XLarge) can only run 10 queries at any given time if a user submits 20 queries all at once to a warehouse with 3X-Large cluster size and cluster scaling (min
1, max1) while 10 queries will start running the remaining 10 queries wait in a queue for these 10 to finish.
5.Increasing the Warehouse cluster size can improve the performance of a query, for example, if a query runs for 1 minute in a 2X-Small warehouse size it may run in 30 Seconds if we change the warehouse size to X-Small. this is due to 2X-Small having 1 worker node and X-Small having 2 worker nodes so the query has more tasks and runs faster (note: this is an ideal case example, the scalability of a query performance depends on many factors, it can not always be linear)
6.A warehouse can have more than one cluster this is called Scale out. If a warehouse is con-figured with X-Small cluster size with cluster scaling(Min1, Max 2) Databricks spins up an additional cluster if it detects queries are waiting in the queue, If a warehouse is configured to run 2 clusters(Min1, Max 2), and let's say a user submits 20 queries, 10 queriers will start running and holds the remaining in the queue and databricks will automatically start the second cluster and starts redirecting the 10 queries waiting in the queue to the second cluster.
7.A single query will not span more than one cluster, once a query is submitted to a cluster it will remain in that cluster until the query execution finishes irrespective of how many clusters are available to scale.
Please review the below diagram to understand the above concepts:
SQL endpoint(SQL Warehouse) scales horizontally(scale-out) and vertical (scale-up), you have to understand when to use what.
Scale-out -> to add more clusters for a SQL endpoint, change max number of clusters If you are trying to improve the throughput, being able to run as many queries as possible then having an additional cluster(s) will improve the performance.
Databricks SQL automatically scales as soon as it detects queries are in queuing state, in this example scaling is set for min 1 and max 3 which means the warehouse can add three clusters if it detects queries are waiting.
During the warehouse creation or after you have the ability to change the warehouse size (2X-Small....to
...4XLarge) to improve query performance and the maximize scaling range to add more clusters on a SQL Endpoint(SQL Warehouse) scale-out, if you are changing an existing warehouse you may have to restart the warehouse to make the changes effective.
How do you know how many clusters you need(How to set Max cluster size)?
When you click on an existing warehouse and select the monitoring tab, you can see warehouse utilization information(see below), there are two graphs that provide important information on how the warehouse is being utilized, if you see queries are being queued that means your warehouse can benefit from additional clusters. Please review the additional DBU cost associated with adding clusters so you can take a well balanced decision between cost and performance.
質問 # 32
A new user who currently does not have access to the catalog or schema is requesting access to the customer table in sales schema, but the customer table contains sensitive information, so you have decided to create view on the table excluding columns that are sensitive and granted access to the view GRANT SELECT ON view_name to [email protected] but when the user tries to query the view, gets the error view does not exist. What is the issue preventing user to access the view and how to fix it?
- A. User requires to be put in a special group that has access to PII data
- B. User requires SELECT on the underlying table
- C. User requires USAGE privilege on Sales schema
- D. User has to be the owner of the view
- E. User needs ADMIN privilege on the view
正解：C
解説：
Explanation
The answer is User requires USAGE privilege on Sales schema,
Data object privileges - Azure Databricks | Microsoft Docs
GRANT USAGE ON SCHEMA sales TO [email protected];
*USAGE: does not give any abilities, but is an additional requirement to perform any action on a schema object.
質問 # 33
A data engineer is overwriting data in a table by deleting the table and recreating the table. Another data
engineer suggests that this is inefficient and the table should simply be overwritten instead.
Which of the following reasons to overwrite the table instead of deleting and recreating the table is incorrect?
- A. Overwriting a table is an atomic operation and will not leave the table in an unfinished state
- B. Overwriting a table allows for concurrent queries to be completed while in progress
- C. Overwriting a table results in a clean table history for logging and audit purposes
- D. Overwriting a table maintains the old version of the table for Time Travel
- E. Overwriting a table is efficient because no files need to be deleted
正解：C
質問 # 34
An external object storage container has been mounted to the location/mnt/finance_eda_bucket.
The following logic was executed to create a database for the finance team:
After the database was successfully created and permissions configured, a member of the finance team runs the following code:
If all users on the finance team are members of thefinancegroup, which statement describes how thetx_sales table will be created?
- A. A managed table will be created in the DBFS root storage container.
- B. An managed table will be created in the storage container mounted to /mnt/finance eda bucket.
- C. A logical table will persist the physical plan to the Hive Metastore in the Databricks control plane.
- D. A logical table will persist the query plan to the Hive Metastore in the Databricks control plane.
- E. An external table will be created in the storage container mounted to /mnt/finance eda bucket.
正解：E
解説：
Explanation
The code uses the CREATE TABLE USING DELTA command to create a Delta Lake table from an existing Parquet file stored in an external object storage container mounted to /mnt/finance_eda_bucket. The code also uses the LOCATION keyword to specify the path to the Parquet file as
/mnt/finance_eda_bucket/tx_sales.parquet. By using the LOCATION keyword, the code creates an external table, which is a table that is stored outside of the default warehouse directory and whose metadata is not managed by Databricks. An external table can be created from an existing directory in a cloud storage system, such as DBFS or S3, that contains data files in a supported format, such as Parquet or CSV. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Create an external table" section.
質問 # 35
Which of the following table constraints that can be enforced on Delta lake tables are supported?
- A. Primary key, foreign key, Not Null, Check Constraints
- B. Not Null, Check Constraints
- C. Default, Not Null, Check Constraints
- D. Unique, Not Null, Check Constraints
- E. Primary key, Not Null, Check Constraints
正解：B
解説：
Explanation
The answer is Not Null, Check Constraints
https://docs.microsoft.com/en-us/azure/databricks/delta/delta-constraints
* CREATE TABLE events( id LONG,
* date STRING,
* location STRING,
* description STRING
* ) USING DELTA;
ALTER TABLE events CHANGE COLUMN id SET NOT NULL;
ALTER TABLE events ADD CONSTRAINT dateWithinRange CHECK (date > '1900-01-01'); Note: Databricks as of DBR 11.1 added support for Primary Key and Foreign Key when Unity Catalog is enabled but this is for information purposes only these are not actually enforced. You may ask then why are we defining these if they are not enforced, so especially these information constraints are very helpful if you have a BI tool that can benefit from knowing the relationship between the tables, so it will be easy when creating reports/dashboards or understanding the data model when using any Data modeling tool.
Primary and Foreign Key
Graphical user interface, text, application, email Description automatically generated
質問 # 36
Which of the following benefits does Delta Live Tables provide for ELT pipelines over standard data pipelines
that utilize Spark and Delta Lake on Databricks?
- A. The ability to declare and maintain data table dependencies
- B. The ability to access previous versions of data tables
- C. The ability to automatically scale compute resources
- D. The ability to write pipelines in Python and/or SQL
- E. The ability to perform batch and streaming queries
正解：A
質問 # 37
Data science team members are using a single cluster to perform data analysis, although cluster size was chosen to handle multiple users and auto-scaling was enabled, the team realized queries are still running slow, what would be the suggested fix for this?
- A. Use High concurrency mode instead of the standard mode
- B. Disable the auto-scaling feature
- C. Increase the size of the driver node
- D. Setup multiple clusters so each team member has their own cluster
正解：A
解説：
Explanation
The answer is Use High concurrency mode instead of the standard mode,
https://docs.databricks.com/clusters/cluster-config-best-practices.html#cluster-mode High Concurrency clusters are ideal for groups of users who need to share resources or run ad-hoc jobs.
Databricks recommends enabling autoscaling for High Concurrency clusters.
質問 # 38
Which of the following locations hosts the driver and worker nodes of a Databricks-managed clus-ter?
- A. JDBC data source
- B. Databricks web application
- C. Control plane
- D. Data plane
- E. Databricks Filesystem
正解：D
解説：
Explanation
The answer is Data Plane, which is where compute(all-purpose, Job Cluster, DLT) are stored this is generally a customer cloud account, there is one exception SQL Warehouses, currently there are 3 types of SQL Warehouse compute available(classic, pro, serverless), in classic and pro compute is located in customer cloud account but serverless computed is located in Databricks cloud account.
Diagram, timeline Description automatically generated
質問 # 39
Which of the following python statements can be used to replace the schema name and table name in the query?
- A. 1.table_name = "sales"
2.query = f"select * from + schema_name +"."+table_name" - B. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = f"select * from schema_name.table_name" - C. 1.table_name = "sales"
2.query = f"select * from {schema_name}.{table_name}" - D. 1.table_name = "sales"
2.query = "select * from {schema_name}.{table_name}"
正解：C
解説：
Explanation
The answer is
1.table_name = "sales"
2.query = f"select * from {schema_name}.{table_name}"
It is always best to use f strings to replace python variables, rather than using string concatenation.
質問 # 40
Which of the following python statement can be used to replace the schema name and table name in the query statement?
- A. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = f"select * from + schema_name +"."+table_name" - B. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = f"select * from schema_name.table_name" - C. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = f"select * from { schema_name}.{table_name}" - D. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = "select * from {schema_name}.{table_name}"
正解：C
解説：
Explanation
Answer is
table_name = "sales"
query = f"select * from {schema_name}.{table_name}"
f strings can be used to format a string. f" This is string {python variable}"
https://realpython.com/python-f-strings/
質問 # 41
Which of the statements is correct when choosing between lakehouse and Datawarehouse?
- A. Lakehouse replaces the current dependency on data lakes and data warehouses uses an open standard storage format and supports low latency BI workloads.
- B. SQL support is only available for Traditional Datawarehouse's, Lakehouses support Python and Scala
- C. Traditional Data warehouses are the preferred choice if we need to support ACID, Lakehouse does not support ACID.
- D. Traditional Data warehouses can serve low query latency with high reliability for BI workloads
- E. Traditional Data warehouses have special indexes which are optimized for Machine learning
正解：A
解説：
Explanation
The lakehouse replaces the current dependency on data lakes and data warehouses for modern data companies that desire:
* Open, direct access to data stored in standard data formats.
* Indexing protocols optimized for machine learning and data science.
* Low query latency and high reliability for BI and advanced analytics.
質問 # 42
What type of table is created when you create delta table with below command?
CREATE TABLE transactions USING DELTA LOCATION "DBFS:/mnt/bronze/transactions"
- A. Managed delta table
- B. Managed table
- C. External table
- D. Delta Lake table
- E. Temp table
正解：C
解説：
Explanation
Anytime a table is created using the LOCATION keyword it is considered an external table, below is the current syntax.
Syntax
CREATE TABLE table_name ( column column_data_type...) USING format LOCATION "dbfs:/" format -> DELTA, JSON, CSV, PARQUET, TEXT I created the table command based on the above question, you can see it created an external table,
質問 # 43
......
Databricksは、組織が大量のデータを迅速かつ効率的に処理することができるクラウドベースのデータエンジニアリングプラットフォームです。このプラットフォームは、Apache Sparkを活用してデータ処理タスクを実行し、データエンジニアリングワークフローを支援するための多数のツールやサービスを提供しています。Databricksは、プラットフォームの使用における専門知識を証明したいデータプロフェッショナル向けの認定プログラムも提供しています。その1つが、Databricks認定プロフェッショナルデータエンジニア試験です。
Databricks-Certified-Professional-Data-Engineer問題集レビュー専門クイズで学習材料：https://www.passtest.jp/Databricks/Databricks-Certified-Professional-Data-Engineer-shiken.html
Databricks-Certified-Professional-Data-Engineerテスト準備トレーニング練習試験問題練習テスト：https://drive.google.com/open?id=1XSC9val4ptWBjSm9gnBndQFV-BGDA9Th