2025年最新の有効なDatabricks-Certified-Professional-Data-Engineerリアル試験問題（更新された）100%問題集と練習試験合格させます [Q30-Q55]

2025年最新の有効なDatabricks-Certified-Professional-Data-Engineerリアル試験問題（更新された）100%問題集と練習試験合格させます

[更新されたのは2025年]Databricks Databricks-Certified-Professional-Data-Engineer問題準備には無料サンプルのPDF

Databricks Certified Professional Data Engineer（Databricks-Certified-Professional-Data-Engineer）試験は、Databricksを使用したデータエンジニアリングソリューションの構築、展開、および保守に関する専門知識を示したい個人向けに設計された認定プログラムです。この試験は、大規模なデータ処理システムを扱うデータエンジニア、データアーキテクト、その他のデータ専門家を対象としており、この分野のスキルと知識を検証することを目的としています。

 

質問 # 30
A Delta table of weather records is partitioned by date and has the below schema:
date DATE, device_id INT, temp FLOAT, latitude FLOAT, longitude FLOAT
To find all the records from within the Arctic Circle, you execute a query with the below filter:
latitude > 66.3
Which statement describes how the Delta engine identifies which files to load?

• A. The Delta log is scanned for min and max statistics for the latitude column
• B. The Parquet file footers are scanned for min and max statistics for the latitude column
• C. All records are cached to attached storage and then the filter is applied
• D. All records are cached to an operational database and then the filter is applied
• E. The Hive metastore is scanned for min and max statistics for the latitude column

正解：A

解説：
Explanation
This is the correct answer because Delta Lake uses a transaction log to store metadata about each table, including min and max statistics for each column in each data file. The Delta engine can use this information to quickly identify which files to load based on a filter condition, without scanning the entire table or the file footers. This is called data skipping and it can improve query performance significantly. Verified References:
[Databricks Certified Data Engineer Professional], under "Delta Lake" section; [Databricks Documentation], under "Optimizations - Data Skipping" section.
In the Transaction log, Delta Lake captures statistics for each data file of the table. These statistics indicate per file:
- Total number of records
- Minimum value in each column of the first 32 columns of the table
- Maximum value in each column of the first 32 columns of the table
- Null value counts for in each column of the first 32 columns of the table When a query with a selective filter is executed against the table, the query optimizer uses these statistics to generate the query result. it leverages them to identify data files that may contain records matching the conditional filter.
For the SELECT query in the question, The transaction log is scanned for min and max statistics for the price column

質問 # 31
A user new to Databricks is trying to troubleshoot long execution times for some pipeline logic they are working on. Presently, the user is executing code cell-by-cell, usingdisplay()calls to confirm code is producing the logically correct results as new transformations are added to an operation. To get a measure of average time to execute, the user is running each cell multiple times interactively.
Which of the following adjustments will get a more accurate measure of how code is likely to perform in production?

• A. Scala is the only language that can be accurately tested using interactive notebooks; because the best performance is achieved by using Scala code compiled to JARs. all PySpark and Spark SQL logic should be refactored.
• B. Production code development should only be done using an IDE; executing code against a local build of open source Spark and Delta Lake will provide the most accurate benchmarks for how code will perform in production.
• C. Calling display () forces a job to trigger, while many transformations will only add to the logical query plan; because of caching, repeated execution of the same logic does not provide meaningful results.
• D. The only way to meaningfully troubleshoot code execution times in development notebooks Is to use production-sized data and production-sized clusters with Run All execution.
• E. The Jobs Ul should be leveraged to occasionally run the notebook as a job and track execution time during incremental code development because Photon can only be enabled on clusters launched for scheduled jobs.

正解：C

解説：
Explanation
This is the correct answer because it explains which of the following adjustments will get a more accurate measure of how code is likely to perform in production. The adjustment is that calling display() forces a job to trigger, while many transformations will only add to the logical query plan; because of caching, repeated execution of the same logic does not provide meaningful results. When developing code in Databricks notebooks, one should be aware of how Spark handles transformations and actions. Transformations are operations that create a new DataFrame or Dataset from an existing one, such as filter, select, or join. Actions are operations that trigger a computation on a DataFrame or Dataset and return a result to the driver program or write it to storage, such as count, show, or save. Calling display() on a DataFrame or Dataset is also an action that triggers a computation and displays the result in a notebook cell. Spark uses lazy evaluation for transformations, which means that they are not executed until an action is called. Spark also uses caching to store intermediate results in memory or disk for faster access in subsequent actions. Therefore, calling display() forces a job to trigger, while many transformations will only add to the logical query plan; because of caching, repeated execution of the same logic does not provide meaningful results. To get a more accurate measure of how code is likely to perform in production, one should avoid calling display() too often or clear the cache before running each cell. Verified References: [Databricks Certified Data Engineer Professional], under "Spark Core" section; Databricks Documentation, under "Lazy evaluation" section; Databricks Documentation, under "Caching" section.

質問 # 32
A dataset has been defined using Delta Live Tables and includes an expectations clause:
1. CONSTRAINT valid_timestamp EXPECT (timestamp > '2020-01-01')
What is the expected behaviour when a batch of data containing data that violates these constraints is
processed?

• A. Records that violate the expectation are added to the target dataset and flagged as in-valid in a field added to the target dataset
• B. Records that violate the expectation cause the job to fail
• C. Records that violate the expectation are added to the target dataset and recorded as invalid in the event log
• D. Records that violate the expectation are dropped from the target dataset and loaded into a quarantine table
• E. Records that violate the expectation are dropped from the target dataset and recorded as invalid in the event log

正解：C

質問 # 33
A data architect has designed a system in which two Structured Streaming jobs will concurrently write to a single bronze Delta table. Each job is subscribing to a different topic from an Apache Kafka source, but they will write data with the same schema. To keep the directory structure simple, a data engineer has decided to nest a checkpoint directory to be shared by both streams.
The proposed directory structure is displayed below:

Which statement describes whether this checkpoint directory structure is valid for the given scenario and why?

• A. No; Delta Lake manages streaming checkpoints in the transaction log.
• B. No; each of the streams needs to have its own checkpoint directory.
• C. Yes; both of the streams can share a single checkpoint directory.
• D. No; only one stream can write to a Delta Lake table.
• E. Yes; Delta Lake supports infinite concurrent writers.

正解：B

解説：
This is the correct answer because checkpointing is a critical feature of Structured Streaming that provides fault tolerance and recovery in case of failures. Checkpointing stores the current state and progress of a streaming query in a reliable storage system, such as DBFS or S3. Each streaming query must have its own checkpoint directory that is unique and exclusive to that query. If two streaming queries share the same checkpoint directory, they will interfere with each other and cause unexpected errors or data loss. Verified References: [Databricks Certified Data Engineer Professional], under "Structured Streaming" section; Databricks Documentation, under "Checkpointing" section.

質問 # 34
Which of the following technologies can be used to identify key areas of text when parsing Spark Driver log4j output?

• A. C++
• B. Scala Datasets
• C. Regex
• D. pyspsark.ml.feature
• E. Julia

正解：C

解説：
Explanation
Regex, or regular expressions, are a powerful way of matching patterns in text. They can be used to identify key areas of text when parsing Spark Driver log4j output, such as the log level, the timestamp, the thread name, the class name, the method name, and the message. Regex can be applied in various languages and frameworks, such as Scala, Python, Java, Spark SQL, and Databricks notebooks. References:
https://docs.databricks.com/notebooks/notebooks-use.html#use-regular-expressions
https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html#using-regular-expressions-in-udfs
https://docs.databricks.com/spark/latest/sparkr/functions/regexp_extract.html
https://docs.databricks.com/spark/latest/sparkr/functions/regexp_replace.html

質問 # 35
A table in the Lakehouse namedcustomer_churn_paramsis used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.
The churn prediction model used by the ML team is fairly stable in production. The team is only interested in making predictions on records that have changed in the past 24 hours.
Which approach would simplify the identification of these changed records?

• A. Calculate the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers before making new predictions; only make predictions on those customers not in the previous predictions.
• B. Apply the churn model to all rows in the customer_churn_params table, but implement logic to perform an upsert into the predictions table that ignores rows where predictions have not changed.
• C. Modify the overwrite logic to include a field populated by calling
  spark.sql.functions.current_timestamp() as data are being written; use this field to identify records written on a particular date.
• D. Replace the current overwrite logic with a merge statement to modify only those records that have changed; write logic to make predictions on the changed records identified by the change data feed.
• E. Convert the batch job to a Structured Streaming job using the complete output mode; configure a Structured Streaming job to read from the customer_churn_params table and incrementally predict against the churn model.

正解：D

解説：
Explanation
The approach that would simplify the identification of the changed records is to replace the current overwrite logic with a merge statement to modify only those records that have changed, and write logic to make predictions on the changed records identified by the change data feed. This approach leverages the Delta Lake features of merge and change data feed, which are designed to handle upserts and track row-level changes in a Delta table12. By using merge, the data engineering team can avoid overwriting the entire table every night, and only update or insert the records that have changed in the source data. By using change data feed, the ML team can easily access the change events that have occurred in the customer_churn_params table, and filter them by operation type (update or insert) and timestamp. This way, they can only make predictions on the records that have changed in the past 24 hours, and avoid re-processing the unchanged records.
The other options are not as simple or efficient as the proposed approach, because:
Option A would require applying the churn model to all rows in the customer_churn_params table, which would be wasteful and redundant. It would also require implementing logic to perform an upsert into the predictions table, which would be more complex than using the merge statement.
Option B would require converting the batch job to a Structured Streaming job, which would involve changing the data ingestion and processing logic. It would also require using the complete output mode, which would output the entire result table every time there is a change in the source data, which would be inefficient and costly.
Option C would require calculating the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers, which would be computationally expensive and prone to errors. It would also require storing and accessing the previous predictions, which would add extra storage and I/O costs.
Option D would require modifying the overwrite logic to include a field populated by calling spark.sql.functions.current_timestamp() as data are being written, which would add extra complexity and overhead to the data engineering job. It would also require using this field to identify records written on a particular date, which would be less accurate and reliable than using the change data feed.
References: Merge, Change data feed

質問 # 36
Which of the following locations in Databricks product architecture hosts jobs/pipelines and queries?

• A. Databricks web application
• B. Databricks Filesystem
• C. Data plane
• D. JDBC data source
• E. Control plane

正解：E

解説：
Explanation
The answer is Control Plane,
Databricks operates most of its services out of a control plane and a data plane, please note serverless features like SQL Endpoint and DLT compute use shared compute in Control pane.
Control Plane: Stored in Databricks Cloud Account
*The control plane includes the backend services that Databricks manages in its own Azure account. Notebook commands and many other workspace configurations are stored in the control plane and encrypted at rest.
Data Plane: Stored in Customer Cloud Account
*The data plane is managed by your Azure account and is where your data resides. This is also where data is processed. You can use Azure Databricks connectors so that your clusters can connect to external data sources outside of your Azure account to ingest data or for storage.
Here is the product architecture diagram highlighted where

質問 # 37
Which of the following is not a privilege in the Unity catalog?

• A. CREATE TABLE
• B. DELETE
• C. MODIFY
• D. EXECUTE
• E. SELECT

正解：B

解説：
Explanation
The Answer is DELETE and UPDATE permissions do not exit, you have to use MODIFY which provides both Update and Delete permissions.
Please note: TABLE ACL privilege types are different from Unity Catalog privilege types, please read the question carefully.
Here is the list of all privileges in Unity Catalog:

Unity Catalog Privileges
https://learn.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/sql-ref-privileges#priv Table ACL privileges
https://learn.microsoft.com/en-us/azure/databricks/security/access-control/table-acls/object-privileges#privileges

質問 # 38
A data ingestion task requires a one-TB JSON dataset to be written out to Parquet with a target part-file size of 512 MB. Because Parquet is being used instead of Delta Lake, built-in file-sizing features such as Auto-Optimize & Auto-Compaction cannot be used.
Which strategy will yield the best performance without shuffling data?

• A. Set spark.sql.files.maxPartitionBytes to 512 MB, ingest the data, execute the narrow transformations, and then write to parquet.
• B. Set spark.sql.adaptive.advisoryPartitionSizeInBytes to 512 MB bytes, ingest the data, execute the narrow transformations, coalesce to 2,048 partitions (1TB*1024*1024/512), and then write to parquet.
• C. Ingest the data, execute the narrow transformations, repartition to 2,048 partitions (1TB* 1024*1024/512), and then write to parquet.
• D. Set spark.sql.shuffle.partitions to 512, ingest the data, execute the narrow transformations, and then write to parquet.
• E. Set spark.sql.shuffle.partitions to 2,048 partitions (1TB*1024*1024/512), ingest the data, execute the narrow transformations, optimize the data by sorting it (which automatically repartitions the data), and then write to parquet.

正解：E

質問 # 39
The data engineer team is configuring environment for development testing, and production before beginning migration on a new data pipeline. The team requires extensive testing on both the code and data resulting from code execution, and the team want to develop and test against similar production data as possible.
A junior data engineer suggests that production data can be mounted to the development testing environments, allowing pre production code to execute against production dat a. Because all users have Admin privileges in the development environment, the junior data engineer has offered to configure permissions and mount this data for the team.
Which statement captures best practices for this situation?

• A. Because access to production data will always be verified using passthrough credentials it is safe to mount data to any Databricks development environment.
• B. In environments where interactive code will be executed, production data should only be accessible with read permissions; creating isolated databases for each environment further reduces risks.
• C. Because delta Lake versions all data and supports time travel, it is not possible for user error or malicious actors to permanently delete production data, as such it is generally safe to mount production data anywhere.
• D. All developer, testing and production code and data should exist in a single unified workspace; creating separate environments for testing and development further reduces risks.

正解：B

解説：
The best practice in such scenarios is to ensure that production data is handled securely and with proper access controls. By granting only read access to production data in development and testing environments, it mitigates the risk of unintended data modification. Additionally, maintaining isolated databases for different environments helps to avoid accidental impacts on production data and systems.
Reference:
Databricks best practices for securing data: https://docs.databricks.com/security/index.html

質問 # 40
Which configuration parameter directly affects the size of a spark-partition upon ingestion of data into Spark?

• A. spark.sql.adaptive.advisoryPartitionSizeInBytes
• B. spark.sql.files.openCostInBytes
• C. spark.sql.adaptive.coalescePartitions.minPartitionNum
• D. spark.sql.files.maxPartitionBytes
• E. spark.sql.autoBroadcastJoinThreshold

正解：D

解説：
Explanation
This is the correct answer because spark.sql.files.maxPartitionBytes is a configuration parameter that directly affects the size of a spark-partition upon ingestion of data into Spark. This parameter configures the maximum number of bytes to pack into a single partition when reading files from file-based sources such as Parquet, JSON and ORC. The default value is 128 MB, which means each partition will be roughly 128 MB in size, unless there are too many small files or only one large file. Verified References: [Databricks Certified Data Engineer Professional], under "Spark Configuration" section; Databricks Documentation, under "Available Properties - spark.sql.files.maxPartitionBytes" section.

質問 # 41
Which statement regarding stream-static joins and static Delta tables is correct?

• A. Each microbatch of a stream-static join will use the most recent version of the static Delta table as of each microbatch.
• B. Each microbatch of a stream-static join will use the most recent version of the static Delta table as of the job's initialization.
• C. Stream-static joins cannot use static Delta tables because of consistency issues.
• D. The checkpoint directory will be used to track updates to the static Delta table.
• E. The checkpoint directory will be used to track state information for the unique keys present in the join.

正解：A

解説：
This is the correct answer because stream-static joins are supported by Structured Streaming when one of the tables is a static Delta table. A static Delta table is a Delta table that is not updated by any concurrent writes, such as appends or merges, during the execution of a streaming query. In this case, each microbatch of a stream-static join will use the most recent version of the static Delta table as of each microbatch, which means it will reflect any changes made to the static Delta table before the start of each microbatch. Verified Reference: [Databricks Certified Data Engineer Professional], under "Structured Streaming" section; Databricks Documentation, under "Stream and static joins" section.

質問 # 42
The DevOps team has configured a production workload as a collection of notebooks scheduled to run daily using the Jobs Ul. A new data engineering hire is onboarding to the team and has requested access to one of these notebooks to review the production logic.
What are the maximum notebook permissions that can be granted to the user without allowing accidental changes to production code or data?

• A. Can Read
• B. Can manage
• C. Can run
• D. Can edit

正解：A

解説：
Granting a user 'Can Read' permissions on a notebook within Databricks allows them to view the notebook's content without the ability to execute or edit it. This level of permission ensures that the new team member can review the production logic for learning or auditing purposes without the risk of altering the notebook's code or affecting production data and workflows. This approach aligns with best practices for maintaining security and integrity in production environments, where strict access controls are essential to prevent unintended modifications.References: Databricks documentation on access control and permissions for notebooks within the workspace (https://docs.databricks.com/security/access-control/workspace-acl.html).

質問 # 43
Which of the following statements describes Delta Lake?

• A. Delta Lake is an open source platform to help manage the complete machine learning lifecycle
• B. Delta Lake is an open format storage layer that delivers reliability, security, and per-formance
• C. Delta Lake is an open format storage layer that processes data
• D. Delta Lake is an open source data storage format for distributed data
• E. Delta Lake is an open source analytics engine used for big data workloads

正解：B

解説：
Explanation
Delta Lake

質問 # 44
Which of the following Structured Streaming queries successfully performs a hop from a Silver to Gold table?

• A. 1.(spark.read.load(rawSalesLocation)
  2. .writeStream
  3. .option("checkpointLocation", checkpointPath)
  4. .outputMode("append")
  5. .table("uncleanedSales") )
• B. 1.(spark.readStream.load(rawSalesLocation)
  2..writeStream
  3..option("checkpointLocation", checkpointPath)
  4..outputMode("append")
  5..table("uncleanedSales") )
• C. 1.(spark.table("sales")
  2..groupBy("store")
  3..agg(sum("sales"))
  4..writeStream
  5..option("checkpointLocation", checkpointPath)
  6..outputMode("complete")
  7..table("aggregatedSales") )
  (Correct)
• D. 1.(spark.table("sales")
  2..writeStream
  3..option("checkpointLocation", checkpointPath)
  4..outputMode("complete")
  5..table("sales") )
• E. 1.(spark.table("sales")
  2..withColumn("avgPrice", col("sales") / col("units"))
  3..writeStream
  4..option("checkpointLocation", checkpointPath)
  5..outputMode("append")
  6..table("cleanedSales") )

正解：C

解説：
Explanation
The answer is
1.(spark.table("sales")
2..groupBy("store")
3..agg(sum("sales"))
4..writeStream
5..option("checkpointLocation", checkpointPath)
6..outputMode("complete")
7..table("aggregatedSales") )
The gold layer is normally used to store aggregated data
Review the below link for more info,
Medallion Architecture - Databricks
Gold Layer:
1. Powers Ml applications, reporting, dashboards, ad hoc analytics
2. Refined views of data, typically with aggregations
3. Reduces strain on production systems
4. Optimizes query performance for business-critical data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
A diagram of a house Description automatically generated with low confidence

質問 # 45
Which statement characterizes the general programming model used by Spark Structured Streaming?

• A. Structured Streaming uses specialized hardware and I/O streams to achieve sub-second latency for data transfer.
• B. Structured Streaming is implemented as a messaging bus and is derived from Apache Kafka.
• C. Structured Streaming leverages the parallel processing of GPUs to achieve highly parallel data throughput.
• D. Structured Streaming relies on a distributed network of nodes that hold incremental state values for cached stages.
• E. Structured Streaming models new data arriving in a data stream as new rows appended to an unbounded table.

正解：E

解説：
Explanation
This is the correct answer because it characterizes the general programming model used by Spark Structured Streaming, which is to treat a live data stream as a table that is being continuously appended. This leads to a new stream processing model that is very similar to a batch processing model, where users can express their streaming computation using the same Dataset/DataFrame API as they would use for static data. The Spark SQL engine will take care of running the streaming query incrementally and continuously and updating the final result as streaming data continues to arrive. Verified References: [Databricks Certified Data Engineer Professional], under "Structured Streaming" section; Databricks Documentation, under "Overview" section.

質問 # 46
You were asked to create a table that can store the below data, orderTime is a timestamp but the finance team when they query this data normally prefer the orderTime in date format, you would like to create a calculated column that can convert the orderTime column timestamp datatype to date and store it, fill in the blank to complete the DDL.

• A. AS DEFAULT (CAST(orderTime as DATE))
• B. Delta lake does not support calculated columns, value should be inserted into the table as part of the ingestion process
• C. GENERATED DEFAULT AS (CAST(orderTime as DATE))
• D. GENERATED ALWAYS AS (CAST(orderTime as DATE))
  Correct)
• E. AS (CAST(orderTime as DATE))

正解：D

解説：
Explanation
The answer is, GENERATED ALWAYS AS (CAST(orderTime as DATE))
https://docs.microsoft.com/en-us/azure/databricks/delta/delta-batch#--use-generated-columns Delta Lake supports generated columns which are a special type of columns whose values are au-tomatically generated based on a user-specified function over other columns in the Delta table. When you write to a table with generated columns and you do not explicitly provide values for them, Delta Lake automatically computes the values.
Note: Databricks also supports partitioning using generated column

質問 # 47
What are the advantages of the Hashing Features?

• A. Easily reverse engineer vectors to determine which original feature mapped to a vector location
• B. Requires the less memory
• C. Less pass through the training data

正解：B、C

解説：
Explanation
SGD-based classifiers avoid the need to predetermine vector size by simply picking a reasonable size and
shoehorning the training data into vectors of that size. This approach is known as feature hashing. The
shoehorning is done by picking one or more locations by using a hash of the name of the variable for
continuous variables or a hash of the variable name and the category name or word for categorical, text*like, or
word-like data.
This hashed feature approach has the distinct advantage of requiring less memory and one less pass through
the training data, but it can make it much harder to reverse engineer vectors to determine which original
feature mapped to a vector location. This is because multiple features may hash to the same location. With
large vectors or with multiple locations per feature, this isn't a problem for accuracy but it can make it hard to
understand what a classifier is doing.
An additional benefit of feature hashing is that the unknown and unbounded vocabularies typical of word-like
variables aren't a problem.

質問 # 48
A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Incremental state information should be maintained for 10 minutes for late-arriving data.
Streaming DataFrame df has the following schema:
"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT"
Code block:

Choose the response that correctly fills in the blank within the code block to complete this task.

• A. awaitArrival("event_time", "10 minutes")
• B. withWatermark("event_time", "10 minutes")
• C. slidingWindow("event_time", "10 minutes")
• D. delayWrite("event_time", "10 minutes")
• E. await("event_time + '10 minutes'")

正解：B

解説：
The correct answer is A. withWatermark("event_time", "10 minutes"). This is because the question asks for incremental state information to be maintained for 10 minutes for late-arriving data. The withWatermark method is used to define the watermark for late data. The watermark is a timestamp column and a threshold that tells the system how long to wait for late data. In this case, the watermark is set to 10 minutes. The other options are incorrect because they are not valid methods or syntax for watermarking in Structured Streaming. Reference:
Watermarking: https://docs.databricks.com/spark/latest/structured-streaming/watermarks.html Windowed aggregations: https://docs.databricks.com/spark/latest/structured-streaming/window-operations.html

質問 # 49
A junior data engineer has ingested a JSON file into a table raw_table with the following schema:
1. cart_id STRING,
2. items ARRAY<item_id:STRING>
The junior data engineer would like to unnest the items column in raw_table to result in a new table with the
following schema:
1.cart_id STRING,
2.item_id STRING
Which of the following commands should the junior data engineer run to complete this task?

• A. 1. SELECT cart_id, flatten(items) AS item_id
  2. FROM raw_table;
• B. 1. SELECT cart_id, explode(items) AS item_id
  2. FROM raw_table;
• C. 1. SELECT cart_id, slice(items) AS item_id
  2. FROM raw_table;
• D. 1. SELECT cart_id, filter(items) AS item_id
  2. FROM raw_table;
• E. 1. SELECT cart_id, reduce(items) AS item_id
  2. FROM raw_table;

正解：B

質問 # 50
A production workload incrementally applies updates from an external Change Data Capture feed to a Delta Lake table as an always-on Structured Stream job. When data was initially migrated for this table, OPTIMIZE was executed and most data files were resized to 1 GB. Auto Optimize and Auto Compaction were both turned on for the streaming production job. Recent review of data files shows that most data files are under 64 MB, although each partition in the table contains at least 1 GB of data and the total table size is over 10 TB.
Which of the following likely explains these smaller file sizes?

• A. Databricks has autotuned to a smaller target file size to reduce duration of MERGE operations
• B. Databricks has autotuned to a smaller target file size based on the overall size of data in the table
• C. Databricks has autotuned to a smaller target file size based on the amount of data in each partition
• D. Z-order indices calculated on the table are preventing file compaction C Bloom filler indices calculated on the table are preventing file compaction

正解：A

解説：
Explanation
This is the correct answer because Databricks has a feature called Auto Optimize, which automatically optimizes the layout of Delta Lake tables by coalescing small files into larger ones and sorting data within each file by a specified column. However, Auto Optimize also considers the trade-off between file size and merge performance, and may choose a smaller target file size to reduce the duration of merge operations, especially for streaming workloads that frequently update existing records. Therefore, it is possible that Auto Optimize has autotuned to a smaller target file size based on the characteristics of the streaming production job. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Auto Optimize" section.

質問 # 51
A Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present, during normal execution, each microbatch of data is processed in less than 3 seconds. During peak hours of the day, execution time for each microbatch becomes very inconsistent, sometimes exceeding 30 seconds. The streaming write is currently configured with a trigger interval of 10 seconds.
Holding all other variables constant and assuming records need to be processed in less than 10 seconds, which adjustment will meet the requirement?

• A. Increase the trigger interval to 30 seconds; setting the trigger interval near the maximum execution time observed for each batch is always best practice to ensure no records are dropped.
• B. Decrease the trigger interval to 5 seconds; triggering batches more frequently may prevent records from backing up and large batches from causing spill.
• C. Decrease the trigger interval to 5 seconds; triggering batches more frequently allows idle executors to begin processing the next batch while longer running tasks from previous batches finish.
• D. Use the trigger once option and configure a Databricks job to execute the query every 10 seconds; this ensures all backlogged records are processed with each batch.
• E. The trigger interval cannot be modified without modifying the checkpoint directory; to maintain the current stream state, increase the number of shuffle partitions to maximize parallelism.

正解：B

解説：
The adjustment that will meet the requirement of processing records in less than 10 seconds is to decrease the trigger interval to 5 seconds. This is because triggering batches more frequently may prevent records from backing up and large batches from causing spill. Spill is a phenomenon where the data in memory exceeds the available capacity and has to be written to disk, which can slow down the processing and increase the execution time1. By reducing the trigger interval, the streaming query can process smaller batches of data more quickly and avoid spill. This can also improve the latency and throughput of the streaming job2.
The other options are not correct, because:
Option A is incorrect because triggering batches more frequently does not allow idle executors to begin processing the next batch while longer running tasks from previous batches finish. In fact, the opposite is true. Triggering batches more frequently may cause concurrent batches to compete for the same resources and cause contention and backpressure2. This can degrade the performance and stability of the streaming job.
Option B is incorrect because increasing the trigger interval to 30 seconds is not a good practice to ensure no records are dropped. Increasing the trigger interval means that the streaming query will process larger batches of data less frequently, which can increase the risk of spill, memory pressure, and timeouts12. This can also increase the latency and reduce the throughput of the streaming job.
Option C is incorrect because the trigger interval can be modified without modifying the checkpoint directory. The checkpoint directory stores the metadata and state of the streaming query, such as the offsets, schema, and configuration3. Changing the trigger interval does not affect the state of the streaming query, and does not require a new checkpoint directory. However, changing the number of shuffle partitions may affect the state of the streaming query, and may require a new checkpoint directory4.
Option D is incorrect because using the trigger once option and configuring a Databricks job to execute the query every 10 seconds does not ensure that all backlogged records are processed with each batch. The trigger once option means that the streaming query will process all the available data in the source and then stop5. However, this does not guarantee that the query will finish processing within 10 seconds, especially if there are a lot of records in the source. Moreover, configuring a Databricks job to execute the query every 10 seconds may cause overlapping or missed batches, depending on the execution time of the query.

質問 # 52
You have written a notebook to generate a summary data set for reporting, Notebook was scheduled using the job cluster, but you realized it takes 8 minutes to start the cluster, what feature can be used to start the cluster in a timely fashion so your job can run immediatley?

• A. Pin the cluster in the cluster UI page so it is always available to the jobs
• B. Use the Databricks cluster pools feature to reduce the startup time
• C. Disable auto termination so the cluster is always running
• D. Use Databricks Premium edition instead of Databricks standard edition
• E. Setup an additional job to run ahead of the actual job so the cluster is running second job starts

正解：B

解説：
Explanation
Cluster pools allow us to reserve VM's ahead of time, when a new job cluster is created VM are grabbed from the pool. Note: when the VM's are waiting to be used by the cluster only cost incurred is Azure. Databricks run time cost is only billed once VM is allocated to a cluster.
Here is a demo of how to setup a pool and follow some best practices,
Graphical user interface, text Description automatically generated

質問 # 53
Consider flipping a coin for which the probability of heads is p, where p is unknown, and our goa is to
estimate p. The obvious approach is to count how many times the coin came up heads and divide by the total
number of coin flips. If we flip the coin 1000 times and it comes up heads 367 times, it is very reasonable to
estimate p as approximately 0.367. However, suppose we flip the coin only twice and we get heads both times.
Is it reasonable to estimate p as 1.0? Intuitively, given that we only flipped the coin twice, it seems a bit
rash to conclude that the coin will always come up heads, and____________is a way of avoiding such rash
conclusions.

• A. Laplace Smoothing
• B. Naive Bayes
• C. Linear Regression
• D. Logistic Regression

正解：A

解説：
Explanation
Smooth the estimates:consider flipping a coin for which the probability of heads is p, where p is unknown, and
our goal is to estimate p. The obvious approach is to count how many times the coin came up heads and divide
by the total number of coin flips. If we flip the coin 1000 times and it comes up heads 367 times, it is very
reasonable to estimate p as approximately 0.367. However, suppose we flip the coin only twice and we get
heads both times. Is it reasonable to estimate p as 1.0? Intuitively, given that we only flipped the coin twice, it
seems a bit rash to conclude that the coin will always come up heads, and smoothing is a way of avoiding such
rash conclusions. A simple smoothing method, called Laplace smoothing (or Laplace's law of succession or
add-one smoothing in R&N), is to estimate p by (one plus the number of heads) / (two plus the total number of
flips). Said differently, if we are keeping count of the number of heads and the number of tails, this rule is
equivalent to starting each of our counts at one, rather than zero. Another advantage of Laplace smoothing is
that it avoids estimating any probabilities to be zero, even for events never observed in the data. Laplace
add-one smoothing now assigns too much probability to unseen words

質問 # 54
You are working on a process to query the table based on batch date, and batch date is an input parameter and expected to change every time the program runs, what is the best way to we can parameterize the query to run without manually changing the batch date?

• A. Store the batch date in the spark configuration and use a spark data frame to filter the data based on the spark configuration.
• B. Create a notebook parameter for batch date and assign the value to a python variable and use a spark data frame to filter the data based on the python variable
• C. There is no way we can combine python variable and spark code
• D. Create a dynamic view that can calculate the batch date automatically and use the view to query the data
• E. Manually edit code every time to change the batch date

正解：B

解説：
Explanation
The answer is, Create a notebook parameter for batch date and assign the value to a python variable and use a spark data frame to filter the data based on the python variable

質問 # 55
......

Databricks Certified Professional Data Engineer試験に合格することは、ビッグデータとクラウドコンピューティングのキャリアを追求している個人にとって重要な成果になる可能性があります。この認定は、候補者がApache Sparkの深い理解を持ち、それを活用してスケーラブルなビッグデータソリューションを設計および実装できることを示しています。さらに、この認定は業界の多くの企業に認められており、候補者が高収入の仕事を得る可能性を高めることができます。

 

Databricks-Certified-Professional-Data-Engineer豪華セット学習ガイドにはオンライン試験エンジン：https://www.passtest.jp/Databricks/Databricks-Certified-Professional-Data-Engineer-shiken.html

2025年最新の認定サンプル問題Databricks-Certified-Professional-Data-Engineer問題集と練習試験：https://drive.google.com/open?id=1d0eFtxylgN5GTqBPeLiaz9fLuLIw6UlS