5.5 Lesson: Data Storage and Persistence

5.5 Data Storage and Persistence

By Klea h

​Spider bots crawl around the web and create metadata: data about the data.

Really big data companies have data centers that hold up to 50 petabytes of data, around 25 trillion pages of text, which is around 179 times the amount of the Library of Congress’ content.

But can we continue indefinitely collecting and storing BIG data, in hopes that we may use it someday?

Structuring can help us utilize this BIG data more efficiently. But it still is not enough.

Multiple Choice

What is the main job of spider bots (also called web crawlers)?

• To design websites

• To create social media posts

• To crawl the web and generate metadata for indexing

• To block harmful websites

Multiple Choice

Why do websites get indexed by spider bots?

• To make the websites easier to hack

• To organize data for faster searches

• To increase ad revenue

• To delete old content

​How storage structures can help?

In a text, it saves information about the amount of time each word appears, in order to compress data. What would this new information accomplish? How is it useful? How can I put it back to its original form?

• Keeps original data safe
  You can change or delete metadata without affecting the original data.

• Helps find and organize info
  Metadata makes it easy to sort or search for specific info, like common words in song lyrics.

• Makes data more useful
  Metadata lets you quickly pull important info, like how often a word appears in many songs.

• Adds structure to data
  Metadata helps you organize and compare data in new ways, like ranking songs by word length.

Multiple Choice

Why is structuring big data important?

• It hides personal information

• It makes data easier to delete

• It helps us use and understand data more efficiently

• It keeps hackers away

Multiple Choice

What is a challenge of continuing to collect Big Data indefinitely?
A. Computers might stop working
B. Data can become too large to manage effectively
C. The internet will slow down
D. Storage space is infinite

• B. Data can become too large to manage effectively
• G. Data can be deleted at any time
• F. All data will be useful
• E. Data collection will always be accurate

​INDEX: One method of metadata is index

Index is an alphabetical list of names and subjects (conceptual topics) with references to the places they occur.

Why do search engines search through indexes of webpages instead of webpages itself when a search query is performed?  

• It is more efficient, and less time consuming. Although it takes time creating an index, you create it once and then reuse for different keywords.
  

• Indexes help make it faster and easier to find specific information.

• Instead of searching through all the data, an index points to where important parts are stored—like a table of contents in a book. This saves time and processing power, which is especially helpful when dealing with huge data sets.

​ORIGINAL DOCUMENT VS ITS METADATA

​Another form of metadata is concordance.

In concordance every word - single or plural is recognized as separate words.

Indexes are like a table of contents — they help you quickly find where data is stored. They list key terms or fields and point to their location in the dataset.

Concordances are more detailed — they show every place a word or phrase appears, often with a bit of context. Think of it like a search with examples.

• Indexes = Fast access to data locations (used for speed and efficiency)

• Concordances = Deeper insight into how terms are used (used for analysis and meaning)

Bible has a concordance, or lawyers use them as well.

Multiple Choice

Why do search engines search through indexes instead of actual webpages?

• Indexes contain full versions of each webpage

• Indexes are hidden from the public

• Indexes are faster and more efficient for finding information

• Indexes are easier to delete

Multiple Choice

What is the key difference between an index and a concordance?
A. Concordances show the exact location of every word with context; indexes list conceptual topics
B. Indexes are used for print books only
C. Indexes are always longer than concordances
D. There is no difference—they are the same

• Concordances show the exact location of every word with context; indexes list conceptual topics

• Indexes are primarily for academic texts; concordances are for fiction

• Concordances are only used in digital formats; indexes are for print

• Indexes are always longer than concordances

• There is no difference—they are the same

Multiple Choice

Which is true about concordances?

• They skip repeated words

• They combine all word forms into one entry

• They are primarily for poetry analysis

• They track every individual word and its location

• They are only used for religious texts

• PII (Personal Identifiable Information) includes:

  • Name, address, phone number

  • Credit card info, medical records, etc.

• We often share PII online (e.g., shopping, signing up for accounts)

​DEACTIVATING VS DELETING ACCOUNTS

​Risks of storing Personal Identifiable Information:

• Can be sold to third parties without your permission

• May be used in harmful or unexpected ways

• Social media dangers:

  • Posts about location or schedules can help criminals

  • Example: Sharing vacation plans could lead to break-ins

• Technology tracks your data:

  • Search engines, websites, and apps track your activity

  • Info like your location, browsing history, and device data is collected

• Be cautious:

  • Think before posting personal details

  • Your data is valuable—and others may try to misuse it

Multiple Choice

What does PII stand for?

• Protected Internet Information

• Private Internet Identity

• Public Info Index

• Personal Identifiable Information

Multiple Choice

Which of the following is an example of sharing PII online?

• Watching a YouTube video

• Liking a meme

• Entering your address on an online store

• Changing your wallpaper

Multiple Choice

What is a possible danger of posting your location on social media?

• You could miss a new post

• You might lose followers

• 
  Your post might not get enough likes

• Criminals might use that info to target you

​Vocabulary (Simplified)

• Relational Database: A way to organize and access data using related tables.

• Generation Loss: Quality loss from copying analog data; doesn't happen with digital if unchanged.

• Browser: A program to access and view websites (e.g., Chrome, Firefox).

• Metadata: Data that describes other data (like info about a photo or webpage).

• Data vs. Information: Data = raw facts; Information = processed and meaningful data.

• Data Persistence: Data that stays saved, even after deletion or long periods of no use.

• Data Storage: Places where data is kept (like CDs, USBs, memory, or tapes).

• Indexing: Organizing data to keep track of it efficiently.

• Filter Bubble: When algorithms limit what info you see based on your past behavior.

• Privacy Concerns: Digital data can be copied, shared, or sold more easily.

• Utility: How useful something is — like trading data for a benefit.

• Cache: Temporary storage for quick access to frequently used data.