Databricks Certified Data Engineer Professional : Databricks-Certified-Data-Engineer-Professional

Which method can be used to determine the total wall-clock time it took to execute a query?

• A. In the Spark UI, take the sum of all task durations that ran across all stages for all jobs associated with that query.
• B. In the Spark UI, take the job duration of the longest-running job associated with that query.
• C. Open the Query Profiler associated with that query and use the Aggregated task time metric.
• D. Open the Query Profiler associated with that query and use the Total wall-clock duration metric.

The data engineering team has configured a job to process customer requests to be forgotten (have their data deleted). All user data that needs to be deleted is stored in Delta Lake tables using default table settings.
The team has decided to process all deletions from the previous week as a batch job at 1am each Sunday. The total duration of this job is less than one hour. Every Monday at 3am, a batch job executes a series of VACUUM commands on all Delta Lake tables throughout the organization.
The compliance officer has recently learned about Delta Lake's time travel functionality. They are concerned that this might allow continued access to deleted data.
Assuming all delete logic is correctly implemented, which statement correctly addresses this concern?

• A. Because the default data retention threshold is 24 hours, data files containing deleted records will be retained until the vacuum job is run the following day.
• B. Because the vacuum command permanently deletes all files containing deleted records, deleted records may be accessible with time travel for around 24 hours.
• C. Because the default data retention threshold is 7 days, data files containing deleted records will be retained until the vacuum job is run 8 days later.
• D. Because Delta Lake's delete statements have ACID guarantees, deleted records will be permanently purged from all storage systems as soon as a delete job completes.
• E. Because Delta Lake time travel provides full access to the entire history of a table, deleted records can always be recreated by users with full admin privileges.

The data governance team has instituted a requirement that the "user" table containing Personal Identifiable Information (PII) must have the appropriate masking on the SSN column. This means that anyone outside of the HRAdminGroup should see masked social security numbers as ***-**-
****.
The team created a masking function:

What does the data governance team need to do next to achieve this goal?

• A. CREATE TABLE users
  (name STRING);
  ALTER TABLE users CREATE COLUMN ssn CREATE MASK ssn_mask;
• B. CREATE TABLE users
  (name STRING, int STRING);
  ALTER TABLE users ALTER COLUMN ssn CREATE MASK if is_member('HRAdminGroup');
• C. CREATE TABLE users
  (name STRING, ssn INT MASKED ssn_mask);
• D. CREATE TABLE users
  (name STRING, ssn STRING);
  ALTER TABLE users ALTER COLUMN ssn SET MASK ssn_mask;

A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Incremental state information should be maintained for 10 minutes for late-arriving data.
Streaming DataFrame df has the following schema:
"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT"
Code block:

Choose the response that correctly fills in the blank within the code block to complete this task.

• A. slidingWindow("event_time", "10 minutes")
• B. await("event_time + `10 minutes'")
• C. awaitArrival("event_time", "10 minutes")
• D. delayWrite("event_time", "10 minutes")
• E. withWatermark("event_time", "10 minutes")

The security team is exploring whether or not the Databricks secrets module can be leveraged for connecting to an external database.
After testing the code with all Python variables being defined with strings, they upload the password to the secrets module and configure the correct permissions for the currently active user. They then modify their code to the following (leaving all other variables unchanged).

Which statement describes what will happen when the above code is executed?

• A. An interactive input box will appear in the notebook; if the right password is provided, the connection will succeed and the encoded password will be saved to DBFS.
• B. The connection to the external table will fail; the string "redacted" will be printed.
• C. The connection to the external table will succeed; the string value of password will be printed in plain text.
• D. The connection to the external table will succeed; the string "redacted" will be printed.
• E. An interactive input box will appear in the notebook; if the right password is provided, the connection will succeed and the password will be printed in plain text.