Dynamic rendering on a layer with a long SQL query

Dynamic rendering on a layer with a long SQL query

I have a question on dynamic rendering with the CartoDB APIs. These APIs are new for me.

Data is a big layer stored in CartoDB (about 30 000 polygons).

In my application the default rendering of this layer must be replaced by a rendering defined by the user. For each subset of selected polygons per the user, the color change. But in a subset the number of polygons can be about 10 000 polygons for example.

In my code I use the subLayerOptions by setting SQL and cartocss properties. The SQL is like "sql:"SELECT * FROM polygons where ID in ('10000','40202'… ".

It works fine with a few values but it doesn't work when the SQL request contains a lot value because the request is too long.

Is there an another method to do this?

In that case you need to do something in the server that group those polygons. Sending a SQL with 10k id's is going to be too large and it will take lot of time to send, process and render.

After you grouped in you could use another where condition like,

select * from polygons where type = 'whatever'

more information would be welcomed

Performance issue with sp_executesql and VARCHAR parameter

Table Segments has an index by DEPARTMENT (VARCHAR(10)) and BDPID(VARCHAR(10)).

Execution time of the first query is 34 seconds

When I moved the parameter for DEPARTMENT to a variable, execution time became 1 second. Execution plan #2 (fast)

But I have to use dynamic sql. And when I moved query to sp_execitesql the execution time again became 34 seconds. Execution plan #3 (slow)

How can I get performance of the second query using dynamic sql?

16 Answers 16

What you are proposing is not new. Plenty of people have tried it. most have found that they chase "infinite" flexibility and instead end up with much, much less than that. It's the "roach motel" of database designs -- data goes in, but it's almost impossible to get it out. Try and conceptualize writing the code for ANY sort of constraint and you'll see what I mean.

The end result typically is a system that is MUCH more difficult to debug, maintain, and full of data consistency problems. This is not always the case, but more often than not, that is how it ends up. Mostly because the programmer(s) don't see this train wreck coming and fail to defensively code against it. Also, often ends up the case that the "infinite" flexibility really isn't that necessary it's a very bad "smell" when the dev team gets a spec that says "Gosh I have no clue what sort of data they are going to put here, so let 'em put WHATEVER". and the end users are just fine having pre-defined attribute types that they can use (code up a generic phone #, and let them create any # of them -- this is trivial in a nicely normalized system and maintains flexibility and integrity!)

If you have a very good development team and are intimately aware of the problems you'll have to overcome with this design, you can successfully code up a well designed, not terribly buggy system. Most of the time.

8 Answers 8

These days, you are likely to see reads (queries) handled differently than writes (commands). In a system with a complicated query, the query itself is unlikely to pass through the domain model (which is primarily responsible for maintaining the consistency of writes).

You are absolutely right that we should render unto SQL that which is SQL. So we'll design a data model optimized around the reads, and a query of that data model will usually take a code path that does not include the domain model (with the possible exception of some input validation -- ensuring that parameters in the query are reasonable).

As I understand it, a main point is to split the Domain Logic (Business Logic) from the Infrastructure (DB, File System, etc.).

This is the foundation of the misunderstanding: the purpose of DDD isn't to separate things along a hard line like "this is in the SQL server, so must not be BL", the purpose of DDD is to separate domains and create barriers between them that allow the internals of a domain to be completely separate from the internals of another domain, and to define shared externals between them.

Don't think of "being in SQL" as the BL/DL barrier&mdashthat's not what it is. Instead, think of "this is the end of the internal domain" as the barrier.

Each domain should have external-facing API's that allow it to work with all the other domains: in the case of the data storage layer, it should have read/write (CRUD) actions for the data-objects it stores. This means SQL itself isn't really the barrier, the VIEW and PROCEDURE components are. You should never read directly from the table: that is the implementation detail DDD tells us that, as an external consumer, we should not worry about.

What I am wondering is, what happens when I have very complex queries like a Material Resource Calculation Query? In that kind of query you work with heavy set operations, the kind of thing that SQL was designed for.

This is exactly what should be in SQL then, and it's not a violation of DDD. It's what we made DDD for. With that calculation in SQL, that becomes part of the BL/DL. What you would do is use a separate view / stored procedure / what-have-you, and keep the business logic separated from the data-layer, as that is your external API. In fact, your data-layer should be another DDD Domain Layer, where your data-layer has it's own abstractions to work with the other domain layers.

Doing these calculations in the infrastructure can't happen too, because the DDD pattern allows for changes in the infrastructure without changing the Domain Layer and knowing that MongoDB doesn't have the same capabilities of e.g. SQL Server, that can't happen.

That's another misunderstanding: it says implementation details internally can change without changing other domain layers. It doesn't say you can just replace a whole infrastructure piece.

Again, keep in mind, DDD is about hiding internals with well-defined external API's. Where those API's sit is a totally different question, and DDD doesn't define that. It simply defines that these API's exist, and should never change.

DDD isn't setup to allow you to ad-hoc replace MSSQL with MongoDB&mdashthose are two totally different infrastructure components.

Instead, let's use an analogy for what DDD defines: gas vs. electric cars. Both of the vehicles have two completely different methods for creating propulsion, but they have the same API's: an on/off, a throttle/brake, and wheels to propel the vehicle. DDD says that we should be able to replace the engine (gas or electric) in our car. It doesn't say we can replace the car with a motorcycle, and that's effectively what MSSQL &rarr MongoDB is.

If you've ever been on a project where the organization paying to host the application decides that the database layer licenses are too expensive, you'll appreciate the ease of which you can migrate your database/data storage. All things considered, while this does happen, it doesn't happen often.

You can get the best of both worlds so to speak. If you consider performing the complex functions in the database an optimization, then you can use an interface to inject an alternate implementation of the calculation. The problem is that you have to maintain logic in multiple locations.

Deviating from an architectural pattern

When you find yourself at odds with implementing a pattern purely, or deviating in some area, then you have a decision to make. A pattern is simply a templated way to do things to help organize your project. At this point take time to evaluate:

  • Is this the right pattern? (many times it is, but sometimes it's just a bad fit)
  • Should I deviate in this one way?
  • Just how far have I deviated so far?

You'll find that some architectural patterns are a good fit for 80-90% of your application, but not so much for the remaining bits. The occasional deviation from the prescribed pattern is useful for performance or logistical reasons.

However, if you find that your cumulative deviations amount to a good deal more than 20% of your application architecture, it's probably just a bad fit.

If you choose to keep going with the architecture, then do yourself a favor and document where and why you deviated from the prescribed way of doing things. When you get a new enthusiastic member on your team, you can point them to that documentation which includes the performance measurements, and justifications. That will reduce the likelihood of repeat requests to fix the "problem". That documentation will also help disincentivize rampant deviations.

The set manipulation logic that SQL is good at can be integrated with DDD no problem.

Say for example I need to know some aggregate value, total count of product by type. Easy to run in sql, but slow if I load every product into memory and add them all up.

I simply introduce a new Domain object,

and a method on my repository

Sure, maybe I am now relying on my DB having certain abilities. But I still technically have the separation and as long as the logic is simple, I can argue that it is not 'business logic'

One of the possible ways to solve this dilemma is to think of SQL as of an assembly language: you rarely, if at all, code directly in it, but where performance matters, you need to be able to understand the code produced by your C/C++/Golang/Rust compiler and maybe even write a tiny snippet in assembly, if you cannot change the code in you high level language to produce desired machine code.

Similarly, in realm of databases and SQL, various SQL libraries (some of which are ORM), e.g. SQLAlchemy and Django ORM for Python, LINQ for .NET, provide higher level abstractions yet use generated SQL code where possible to achieve performance. They also provide some portability as to the used DB, possibly having different performance, e.g. on Postgres and MySQL, due to some operations using some more optimal DB-specific SQL.

And just as with high level languages, it is critical to understand how SQL works, even if it is just to rearrange the queries done with above mentioned SQL libraries, to be able to achieve desired efficiency.

P.S. I would rather make this a comment but I do not have sufficient reputation for that.

As usual, this is one of those things that depends on a number of factors. It's true that there's a lot that you can do with SQL. There are also challenges with using it and some practical limitations of relational databases.

As Jared Goguen notes in the comments, SQL can be very difficult to test and verify. The main factors that lead to this are that it can't (in general) be decomposed into components. In practice, a complex query must be considered in toto. Another complicating factor is that be behavior and correctness of SQL is highly dependent on the structure and content of your data. This means that testing all the possible scenarios (or even determining what they are) is often infeasible or impossible. Refactoring of SQL and modification of database structure is likewise problematic.

The other big factor that has lead to moving away from SQL is relational databases tend to only scale vertically. For example, when you build complex calculations in SQL to run in SQL Server, they are going to execute on the database. That means all of that work is using resources on the database. The more that you do in SQL, the more resources your database will need both in terms of memory and CPU. It's often less efficient to do these things on other systems but there's no practical limit to the number of additional machines you can add to such a solution. This approach is less expensive and more fault-tolerant than building a monster database server.

These issues may or may not apply to the problem at hand. If you are able to solve your problem with available database resources, maybe SQL is fine for your problem-space. You need to consider growth, however. It might be fine today but a few years down the road, the cost of adding additional resources may become a problem.

Is that a pitfall of the DDD pattern?

Let me first clear a few misconceptions.

DDD is not a pattern. And it doesn't really prescribe patterns.

The preface to Eric Evan's DDD book states:

Leading software designers have recognized domain modeling and design as critical topics for at least 20 years, yet surprisingly little has been written about what needs to be done or how to do it. Although it has never been formulated clearly, a philosophy has emerged as an undercurrent in the object community, a philosophy I call domain-driven design.

[. ]

A feature common to the successes was a rich domain model that evolved through iterations of design and became part of the fabric of the project.

This book provides a framework for making design decisions and a technical vocabulary for discussing domain design. It is a synthesis of widely accepted best practices along with my own insights and experiences.

So, it's a way to approach software development and domain modeling, plus some technical vocabulary that supports those activities (a vocabulary that includes various concepts and patterns). It's also not something completely new.

Another thing to keep in mind is that a domain model is not the OO implementation of it that can be found in your system - that's just one way to express it, or to express some part of it. A domain model is the way you think about the problem you are trying to solve with the software. It's how you understand and perceive things, how you talk about them. It's conceptual. But not in some vague sense. It's deep and refined, and is a result of hard work and knowledge gathering. It is further refined and likely evolved over time, and it involves implementation considerations (some of which may constrain the model). It should be shared by all team members (and involved domain experts), and it should drive how you implement the system, so that the system closely reflects it.

Nothing about that is inherently pro- or anti-SQL, although OO developers are perhaps generally better at expressing the model in OO languages, and the expression of many domain concepts is better supported by OOP. But sometimes parts of the model must be expressed in a different paradigm.

What I am wondering is, what happens when I have very complex queries [. ]?

Well, generally speaking there are two scenarios here.

In the first case, some aspect of a domain really requires a complex query, and perhaps that aspect is best expressed in the SQL/relational paradigm - so use the appropriate tool for the job. Reflect those aspects in your domain thinking and the language used in communicating concepts. If the domain is complex, perhaps this is a part of a subdomain with it's own bounded context.

The other scenario is that the perceived need to express something in SQL is a result of constrained thinking. If a person or a team has always been database oriented in their thinking, it may be difficult for them, just due to inertia, to see a different way of approaching things. This becomes a problem when the old way fails to meet the new needs, and requires some thinking out of the box. DDD, as an approach to design, is in part about ways to find your way out of that box by gathering and distilling the knowledge about the domain. But everybody seems to ignore that part of the book, and focuses on some of the technical vocabulary and patterns listed.

Initiating discovery outside of a site configuration

  1. Click the Administration icon.
  2. On the Administration page, under Discovery Options, click the create link for Connections. The Security Console displays the General page of the Asset Discovery Connection panel.
  3. Select the appropriate discovery connection name from the drop-down list labeled Connection.
  4. Click Discover Assets.

With new, changed, or reactivated discovery connections, the discovery process must complete before new discovery results become available. There may be a slight delay before new results appear in the Web interface.

InsightVM establishes the connection and performs discovery. A table appears and lists the following information about each discovered asset.

Displayed values for discovered assets

For mobile devices, the table includes the following:

  • the operating system of the mobile device
  • the account user name for the mobile device
  • the last time the device synced with the Exchange server (WinRM/PowerShell and WinRM/Office 365 only)

For AWS connections, the table includes the following:

  • the name of the AWS instance (asset)
  • the instance's IP address
  • the instance ID
  • the instance's Availability Zone, which is a location within a geographic region that is insulated from failures in other Availability Zones and provides low-latency network connectivity to other Availability Zones in the same region
  • the instance's geographic region
  • the instance type, which defines its memory, CPU, storage capacity, and hourly cost
  • the instance's operating system
  • the operational state of the instance

For VMware connections, the table includes the following:

  • the asset’s name
  • the asset’s IP address
  • the VMware datacenter in which the asset is managed
  • the asset’s host computer
  • the cluster to which the asset belongs
  • the resource pool path that supports the asset
  • the asset’s operating system
  • the asset’s power status

For DHCP connections, the table includes the following:

After performing the initial discovery, the application continues to discover assets as long as the discovery connection remains active. The Security Console displays a notification of any inactive discovery connections in the bar at the top of the Security Console Web interface. You can also check the status of all discovery connections on the Discovery Connections page. See Creating and managing Dynamic Discovery connections .

If you create a discovery connection but don’t initiate discovery with that connection, or if you initiate a discovery but the connection becomes inactive, you will see an advisory icon in the top, left corner of the Web interface page. Roll over the icon to see a message about inactive connections. The message includes a link that you can click to initiate discovery.

After InsightVM discovers assets, they also appear in the Discovered by Connection table on the Assets page. See Locating and working with assets for more information.

Creating a connection in a site configuration

Only the connections listed below can be created within a site configuration.

If you want to create a connection while configuring a new site, click the Create site button on the Home page. OR Click the Create tab at the top of the page and then select Site from the drop-down list.

If you want to create a connection for an existing site, click that site's Edit icon in the Sites table on the Home page.

  1. Click the Assets link in the site configuration .
  2. Select Connection as the option for specifying assets.
  3. Click Create Connection.
  4. Select a connection type:
  • Exchange ActiveSync (LDAP) is for mobile devices managed by an Active Directory (AD) server.
  • Exchange ActiveSync (WinRM/PowerShell) is for mobile devices managed by an on-premises Exchange server accessed with PowerShell.
  • Exchange ActiveSync (WinRM/Office 365) is for mobile devices managed by Cloud-based Exchange server running Microsoft Office 365.
  • vmware vSphere is for environments managed by VMware vCenter or ESX/ESXi.
  • DHCP Service is for assets that Scan Engines discover by collecting log data from DHCP servers.

Adding an Exchange ActiveSync (LDAP) Connection

ActiveSync connections require a dynamic site.

  1. Enter a unique name for the new connection on the New connection tab.
  2. Enter the name of the Active Directory (AD) server to which the Security Console will connect.
  3. Select a protocol from the drop-down list. LDAPS, which is LDAP over SSL, is the more secure option and is recommended if it is enabled on your AD server.
  4. Enter a user name and password for a member of the Organization Management Security Group in Microsoft Exchange. This account will enable the Security Console to discover mobile devices connected to the AD server. Note: Once you save credentials in this setting, changes will not take effect until you restart InsightVM.
  5. Click Save. The connection appears in the Connection drop-down list, which you can view by clicking Select Connection.
  6. Continue with Initiating Dynamic Discovery.

Adding an Exchange ActiveSync (WinRM/PowerShell or WinRM/Office 365) Connection

ActiveSync connections require a dynamic site.

  1. Enter a unique name for the new connection on the New connection tab.
  2. Enter the name of the of the WinRM gateway server to which the Security Console will connect.
  3. Enter a user name and password for an account that has WinRM permissions for the gateway server.
  4. Enter the fully qualified domain name of the Exchange server that manages the mobile device information.
  5. Enter a user name and password for an administrator account or a user account that has View-Only Organizational Management or higher role of the Organization Management Security Group in Microsoft Exchange.
  6. Click Save. The connection appears in the Connection drop-down list, which you can view by clicking Select Connection.
  7. Continue with Initiating Dynamic Discovery.

Adding a VMware vSphere Connection (site configuration)

VMware vSphere connections require a dynamic site.

  1. Enter a unique name for the new connection on the New connection tab.
  2. Enter a fully qualified domain name for the server that the Security Console will contact in order to discover assets.
  3. Enter a port number and select the protocol for the connection.
  4. Enter a user name and password with which the Security Console will log on to the server. Make sure that the account has access to any virtual machine that you want to discover.
  5. Click Save. The connection appears in the Connection drop-down list, which you can view by clicking Select Connection.
  6. Continue with Initiating Dynamic Discovery

Adding a DHCP-Directory Watcher Connection (site configuration)

DHCP connections require a dynamic site.

  1. Enter a unique name for the new connection.
  2. Select an event source type.
  3. Select the Directory Watcher collection method.
  4. Enter a network path to the folder containing the DHCP server logs to be queried. Use the format //server/path/to/folder. The server can be either a host name or IP address.
  5. Select the Scan Engine that will collect the DHCP server log information.
  6. Enter the administrative user name and password for accessing the DHCP server.
  7. Click Save. The connection appears in the Connection drop-down list, which you can view by clicking Select Connection.
  8. Continue with Initiating Dynamic Discovery.

Adding a DHCP-Syslog Connection (site configuration)

DHCP connections require a dynamic site.

Syslog is the only available collection method for the Infoblox Trinzic event source.


Formally, a "database" refers to a set of related data and the way it is organized. Access to this data is usually provided by a "database management system" (DBMS) consisting of an integrated set of computer software that allows users to interact with one or more databases and provides access to all of the data contained in the database (although restrictions may exist that limit access to particular data). The DBMS provides various functions that allow entry, storage and retrieval of large quantities of information and provides ways to manage how that information is organized.

Because of the close relationship between them, the term "database" is often used casually to refer to both a database and the DBMS used to manipulate it.

Outside the world of professional information technology, the term database is often used to refer to any collection of related data (such as a spreadsheet or a card index) as size and usage requirements typically necessitate use of a database management system. [1]

Existing DBMSs provide various functions that allow management of a database and its data which can be classified into four main functional groups:

  • Data definition – Creation, modification and removal of definitions that define the organization of the data.
  • Update – Insertion, modification, and deletion of the actual data. [2]
  • Retrieval – Providing information in a form directly usable or for further processing by other applications. The retrieved data may be made available in a form basically the same as it is stored in the database or in a new form obtained by altering or combining existing data from the database. [3]
  • Administration – Registering and monitoring users, enforcing data security, monitoring performance, maintaining data integrity, dealing with concurrency control, and recovering information that has been corrupted by some event such as an unexpected system failure. [4]

Both a database and its DBMS conform to the principles of a particular database model. [5] "Database system" refers collectively to the database model, database management system, and database. [6]

Physically, database servers are dedicated computers that hold the actual databases and run only the DBMS and related software. Database servers are usually multiprocessor computers, with generous memory and RAID disk arrays used for stable storage. Hardware database accelerators, connected to one or more servers via a high-speed channel, are also used in large volume transaction processing environments. DBMSs are found at the heart of most database applications. DBMSs may be built around a custom multitasking kernel with built-in networking support, but modern DBMSs typically rely on a standard operating system to provide these functions. [ citation needed ]

Since DBMSs comprise a significant market, computer and storage vendors often take into account DBMS requirements in their own development plans. [7]

Databases and DBMSs can be categorized according to the database model(s) that they support (such as relational or XML), the type(s) of computer they run on (from a server cluster to a mobile phone), the query language(s) used to access the database (such as SQL or XQuery), and their internal engineering, which affects performance, scalability, resilience, and security.

The sizes, capabilities, and performance of databases and their respective DBMSs have grown in orders of magnitude. These performance increases were enabled by the technology progress in the areas of processors, computer memory, computer storage, and computer networks. The concept of a database was made possible by the emergence of direct access storage media such as magnetic disks, which became widely available in the mid 1960s earlier systems relied on sequential storage of data on magnetic tape. The subsequent development of database technology can be divided into three eras based on data model or structure: navigational, [8] SQL/relational, and post-relational.

The two main early navigational data models were the hierarchical model and the CODASYL model (network model). These were characterized by the use of pointers (often physical disk addresses) to follow relationships from one record to another.

The relational model, first proposed in 1970 by Edgar F. Codd, departed from this tradition by insisting that applications should search for data by content, rather than by following links. The relational model employs sets of ledger-style tables, each used for a different type of entity. Only in the mid-1980s did computing hardware become powerful enough to allow the wide deployment of relational systems (DBMSs plus applications). By the early 1990s, however, relational systems dominated in all large-scale data processing applications, and as of 2018 [update] they remain dominant: IBM DB2, Oracle, MySQL, and Microsoft SQL Server are the most searched DBMS. [9] The dominant database language, standardised SQL for the relational model, has influenced database languages for other data models. [ citation needed ]

Object databases were developed in the 1980s to overcome the inconvenience of object–relational impedance mismatch, which led to the coining of the term "post-relational" and also the development of hybrid object–relational databases.

The next generation of post-relational databases in the late 2000s became known as NoSQL databases, introducing fast key–value stores and document-oriented databases. A competing "next generation" known as NewSQL databases attempted new implementations that retained the relational/SQL model while aiming to match the high performance of NoSQL compared to commercially available relational DBMSs.

1960s, navigational DBMS

The introduction of the term database coincided with the availability of direct-access storage (disks and drums) from the mid-1960s onwards. The term represented a contrast with the tape-based systems of the past, allowing shared interactive use rather than daily batch processing. The Oxford English Dictionary cites a 1962 report by the System Development Corporation of California as the first to use the term "data-base" in a specific technical sense. [10]

As computers grew in speed and capability, a number of general-purpose database systems emerged by the mid-1960s a number of such systems had come into commercial use. Interest in a standard began to grow, and Charles Bachman, author of one such product, the Integrated Data Store (IDS), founded the Database Task Group within CODASYL, the group responsible for the creation and standardization of COBOL. In 1971, the Database Task Group delivered their standard, which generally became known as the CODASYL approach, and soon a number of commercial products based on this approach entered the market.

The CODASYL approach offered applications the ability to navigate around a linked data set which was formed into a large network. Applications could find records by one of three methods:

  1. Use of a primary key (known as a CALC key, typically implemented by hashing)
  2. Navigating relationships (called sets) from one record to another
  3. Scanning all the records in a sequential order

Later systems added B-trees to provide alternate access paths. Many CODASYL databases also added a declarative query language for end users (as distinct from the navigational API). However CODASYL databases were complex and required significant training and effort to produce useful applications.

IBM also had their own DBMS in 1966, known as Information Management System (IMS). IMS was a development of software written for the Apollo program on the System/360. IMS was generally similar in concept to CODASYL, but used a strict hierarchy for its model of data navigation instead of CODASYL's network model. Both concepts later became known as navigational databases due to the way data was accessed: the term was popularized by Bachman's 1973 Turing Award presentation The Programmer as Navigator. IMS is classified by IBM as a hierarchical database. IDMS and Cincom Systems' TOTAL database are classified as network databases. IMS remains in use as of 2014 [update] . [11]

1970s, relational DBMS

Edgar F. Codd worked at IBM in San Jose, California, in one of their offshoot offices that was primarily involved in the development of hard disk systems. He was unhappy with the navigational model of the CODASYL approach, notably the lack of a "search" facility. In 1970, he wrote a number of papers that outlined a new approach to database construction that eventually culminated in the groundbreaking A Relational Model of Data for Large Shared Data Banks. [12]

In this paper, he described a new system for storing and working with large databases. Instead of records being stored in some sort of linked list of free-form records as in CODASYL, Codd's idea was to organize the data as a number of "tables", each table being used for a different type of entity. Each table would contain a fixed number of columns containing the attributes of the entity. One or more columns of each table were designated as a primary key by which the rows of the table could be uniquely identified cross-references between tables always used these primary keys, rather than disk addresses, and queries would join tables based on these key relationships, using a set of operations based on the mathematical system of relational calculus (from which the model takes its name). Splitting the data into a set of normalized tables (or relations) aimed to ensure that each "fact" was only stored once, thus simplifying update operations. Virtual tables called views could present the data in different ways for different users, but views could not be directly updated.

Codd used mathematical terms to define the model: relations, tuples, and domains rather than tables, rows, and columns. The terminology that is now familiar came from early implementations. Codd would later criticize the tendency for practical implementations to depart from the mathematical foundations on which the model was based.

The use of primary keys (user-oriented identifiers) to represent cross-table relationships, rather than disk addresses, had two primary motivations. From an engineering perspective, it enabled tables to be relocated and resized without expensive database reorganization. But Codd was more interested in the difference in semantics: the use of explicit identifiers made it easier to define update operations with clean mathematical definitions, and it also enabled query operations to be defined in terms of the established discipline of first-order predicate calculus because these operations have clean mathematical properties, it becomes possible to rewrite queries in provably correct ways, which is the basis of query optimization. There is no loss of expressiveness compared with the hierarchic or network models, though the connections between tables are no longer so explicit.

In the hierarchic and network models, records were allowed to have a complex internal structure. For example, the salary history of an employee might be represented as a "repeating group" within the employee record. In the relational model, the process of normalization led to such internal structures being replaced by data held in multiple tables, connected only by logical keys.

For instance, a common use of a database system is to track information about users, their name, login information, various addresses and phone numbers. In the navigational approach, all of this data would be placed in a single variable-length record. In the relational approach, the data would be normalized into a user table, an address table and a phone number table (for instance). Records would be created in these optional tables only if the address or phone numbers were actually provided.

As well as identifying rows/records using logical identifiers rather than disk addresses, Codd changed the way in which applications assembled data from multiple records. Rather than requiring applications to gather data one record at a time by navigating the links, they would use a declarative query language that expressed what data was required, rather than the access path by which it should be found. Finding an efficient access path to the data became the responsibility of the database management system, rather than the application programmer. This process, called query optimization, depended on the fact that queries were expressed in terms of mathematical logic.

Codd's paper was picked up by two people at Berkeley, Eugene Wong and Michael Stonebraker. They started a project known as INGRES using funding that had already been allocated for a geographical database project and student programmers to produce code. Beginning in 1973, INGRES delivered its first test products which were generally ready for widespread use in 1979. INGRES was similar to System R in a number of ways, including the use of a "language" for data access, known as QUEL. Over time, INGRES moved to the emerging SQL standard.

IBM itself did one test implementation of the relational model, PRTV, and a production one, Business System 12, both now discontinued. Honeywell wrote MRDS for Multics, and now there are two new implementations: Alphora Dataphor and Rel. Most other DBMS implementations usually called relational are actually SQL DBMSs.

In 1970, the University of Michigan began development of the MICRO Information Management System [13] based on D.L. Childs' Set-Theoretic Data model. [14] [15] [16] MICRO was used to manage very large data sets by the US Department of Labor, the U.S. Environmental Protection Agency, and researchers from the University of Alberta, the University of Michigan, and Wayne State University. It ran on IBM mainframe computers using the Michigan Terminal System. [17] The system remained in production until 1998.

Integrated approach

In the 1970s and 1980s, attempts were made to build database systems with integrated hardware and software. The underlying philosophy was that such integration would provide higher performance at a lower cost. Examples were IBM System/38, the early offering of Teradata, and the Britton Lee, Inc. database machine.

Another approach to hardware support for database management was ICL's CAFS accelerator, a hardware disk controller with programmable search capabilities. In the long term, these efforts were generally unsuccessful because specialized database machines could not keep pace with the rapid development and progress of general-purpose computers. Thus most database systems nowadays are software systems running on general-purpose hardware, using general-purpose computer data storage. However, this idea is still pursued for certain applications by some companies like Netezza and Oracle (Exadata).

Late 1970s, SQL DBMS

IBM started working on a prototype system loosely based on Codd's concepts as System R in the early 1970s. The first version was ready in 1974/5, and work then started on multi-table systems in which the data could be split so that all of the data for a record (some of which is optional) did not have to be stored in a single large "chunk". Subsequent multi-user versions were tested by customers in 1978 and 1979, by which time a standardized query language – SQL [ citation needed ] – had been added. Codd's ideas were establishing themselves as both workable and superior to CODASYL, pushing IBM to develop a true production version of System R, known as SQL/DS, and, later, Database 2 (DB2).

Larry Ellison's Oracle Database (or more simply, Oracle) started from a different chain, based on IBM's papers on System R. Though Oracle V1 implementations were completed in 1978, it wasn't until Oracle Version 2 when Ellison beat IBM to market in 1979. [18]

Stonebraker went on to apply the lessons from INGRES to develop a new database, Postgres, which is now known as PostgreSQL. PostgreSQL is often used for global mission-critical applications (the .org and .info domain name registries use it as their primary data store, as do many large companies and financial institutions).

In Sweden, Codd's paper was also read and Mimer SQL was developed from the mid-1970s at Uppsala University. In 1984, this project was consolidated into an independent enterprise.

Another data model, the entity–relationship model, emerged in 1976 and gained popularity for database design as it emphasized a more familiar description than the earlier relational model. Later on, entity–relationship constructs were retrofitted as a data modeling construct for the relational model, and the difference between the two have become irrelevant. [ citation needed ]

1980s, on the desktop

The 1980s ushered in the age of desktop computing. The new computers empowered their users with spreadsheets like Lotus 1-2-3 and database software like dBASE. The dBASE product was lightweight and easy for any computer user to understand out of the box. C. Wayne Ratliff, the creator of dBASE, stated: "dBASE was different from programs like BASIC, C, FORTRAN, and COBOL in that a lot of the dirty work had already been done. The data manipulation is done by dBASE instead of by the user, so the user can concentrate on what he is doing, rather than having to mess with the dirty details of opening, reading, and closing files, and managing space allocation." [19] dBASE was one of the top selling software titles in the 1980s and early 1990s.

1990s, object-oriented

The 1990s, along with a rise in object-oriented programming, saw a growth in how data in various databases were handled. Programmers and designers began to treat the data in their databases as objects. That is to say that if a person's data were in a database, that person's attributes, such as their address, phone number, and age, were now considered to belong to that person instead of being extraneous data. This allows for relations between data to be relations to objects and their attributes and not to individual fields. [20] The term "object–relational impedance mismatch" described the inconvenience of translating between programmed objects and database tables. Object databases and object–relational databases attempt to solve this problem by providing an object-oriented language (sometimes as extensions to SQL) that programmers can use as alternative to purely relational SQL. On the programming side, libraries known as object–relational mappings (ORMs) attempt to solve the same problem.

2000s, NoSQL and NewSQL

XML databases are a type of structured document-oriented database that allows querying based on XML document attributes. XML databases are mostly used in applications where the data is conveniently viewed as a collection of documents, with a structure that can vary from the very flexible to the highly rigid: examples include scientific articles, patents, tax filings, and personnel records.

NoSQL databases are often very fast, do not require fixed table schemas, avoid join operations by storing denormalized data, and are designed to scale horizontally.

In recent years, there has been a strong demand for massively distributed databases with high partition tolerance, but according to the CAP theorem it is impossible for a distributed system to simultaneously provide consistency, availability, and partition tolerance guarantees. A distributed system can satisfy any two of these guarantees at the same time, but not all three. For that reason, many NoSQL databases are using what is called eventual consistency to provide both availability and partition tolerance guarantees with a reduced level of data consistency.

NewSQL is a class of modern relational databases that aims to provide the same scalable performance of NoSQL systems for online transaction processing (read-write) workloads while still using SQL and maintaining the ACID guarantees of a traditional database system.

Databases are used to support internal operations of organizations and to underpin online interactions with customers and suppliers (see Enterprise software).

Databases are used to hold administrative information and more specialized data, such as engineering data or economic models. Examples include computerized library systems, flight reservation systems, computerized parts inventory systems, and many content management systems that store websites as collections of webpages in a database.

One way to classify databases involves the type of their contents, for example: bibliographic, document-text, statistical, or multimedia objects. Another way is by their application area, for example: accounting, music compositions, movies, banking, manufacturing, or insurance. A third way is by some technical aspect, such as the database structure or interface type. This section lists a few of the adjectives used to characterize different kinds of databases.

  • An in-memory database is a database that primarily resides in main memory, but is typically backed-up by non-volatile computer data storage. Main memory databases are faster than disk databases, and so are often used where response time is critical, such as in telecommunications network equipment.
  • An active database includes an event-driven architecture which can respond to conditions both inside and outside the database. Possible uses include security monitoring, alerting, statistics gathering and authorization. Many databases provide active database features in the form of database triggers.
  • A cloud database relies on cloud technology. Both the database and most of its DBMS reside remotely, "in the cloud", while its applications are both developed by programmers and later maintained and used by end-users through a web browser and Open APIs. archive data from operational databases and often from external sources such as market research firms. The warehouse becomes the central source of data for use by managers and other end-users who may not have access to operational data. For example, sales data might be aggregated to weekly totals and converted from internal product codes to use UPCs so that they can be compared with ACNielsen data. Some basic and essential components of data warehousing include extracting, analyzing, and mining data, transforming, loading, and managing data so as to make them available for further use.
  • A deductive database combines logic programming with a relational database.
  • A distributed database is one in which both the data and the DBMS span multiple computers.
  • A document-oriented database is designed for storing, retrieving, and managing document-oriented, or semi structured, information. Document-oriented databases are one of the main categories of NoSQL databases.
  • An embedded database system is a DBMS which is tightly integrated with an application software that requires access to stored data in such a way that the DBMS is hidden from the application's end-users and requires little or no ongoing maintenance. [21]
  • End-user databases consist of data developed by individual end-users. Examples of these are collections of documents, spreadsheets, presentations, multimedia, and other files. Several products exist to support such databases. Some of them are much simpler than full-fledged DBMSs, with more elementary DBMS functionality.
  • A federated database system comprises several distinct databases, each with its own DBMS. It is handled as a single database by a federated database management system (FDBMS), which transparently integrates multiple autonomous DBMSs, possibly of different types (in which case it would also be a heterogeneous database system), and provides them with an integrated conceptual view.
  • Sometimes the term multi-database is used as a synonym to federated database, though it may refer to a less integrated (e.g., without an FDBMS and a managed integrated schema) group of databases that cooperate in a single application. In this case, typically middleware is used for distribution, which typically includes an atomic commit protocol (ACP), e.g., the two-phase commit protocol, to allow distributed (global) transactions across the participating databases.
  • A graph database is a kind of NoSQL database that uses graph structures with nodes, edges, and properties to represent and store information. General graph databases that can store any graph are distinct from specialized graph databases such as triplestores and network databases.
  • An array DBMS is a kind of NoSQL DBMS that allows modeling, storage, and retrieval of (usually large) multi-dimensional arrays such as satellite images and climate simulation output.
  • In a hypertext or hypermedia database, any word or a piece of text representing an object, e.g., another piece of text, an article, a picture, or a film, can be hyperlinked to that object. Hypertext databases are particularly useful for organizing large amounts of disparate information. For example, they are useful for organizing online encyclopedias, where users can conveniently jump around the text. The World Wide Web is thus a large distributed hypertext database.
  • A knowledge base (abbreviated KB, kb or Δ [22][23] ) is a special kind of database for knowledge management, providing the means for the computerized collection, organization, and retrieval of knowledge. Also a collection of data representing problems with their solutions and related experiences.
  • A mobile database can be carried on or synchronized from a mobile computing device. store detailed data about the operations of an organization. They typically process relatively high volumes of updates using transactions. Examples include customer databases that record contact, credit, and demographic information about a business's customers, personnel databases that hold information such as salary, benefits, skills data about employees, enterprise resource planning systems that record details about product components, parts inventory, and financial databases that keep track of the organization's money, accounting and financial dealings.
  • A parallel database seeks to improve performance through parallelization for tasks such as loading data, building indexes and evaluating queries.
  • Shared memory architecture, where multiple processors share the main memory space, as well as other data storage.
  • Shared disk architecture, where each processing unit (typically consisting of multiple processors) has its own main memory, but all units share the other storage.
  • Shared-nothing architecture, where each processing unit has its own main memory and other storage.
    employ fuzzy logic to draw inferences from imprecise data. process transactions fast enough for the result to come back and be acted on right away.
  • A spatial database can store the data with multidimensional features. The queries on such data include location-based queries, like "Where is the closest hotel in my area?".
  • A temporal database has built-in time aspects, for example a temporal data model and a temporal version of SQL. More specifically the temporal aspects usually include valid-time and transaction-time.
  • A terminology-oriented database builds upon an object-oriented database, often customized for a specific field.
  • An unstructured data database is intended to store in a manageable and protected way diverse objects that do not fit naturally and conveniently in common databases. It may include email messages, documents, journals, multimedia objects, etc. The name may be misleading since some objects can be highly structured. However, the entire possible object collection does not fit into a predefined structured framework. Most established DBMSs now support unstructured data in various ways, and new dedicated DBMSs are emerging.

Connolly and Begg define database management system (DBMS) as a "software system that enables users to define, create, maintain and control access to the database". [24] Examples of DBMS's include MySQL, PostgreSQL, Microsoft SQL Server, Oracle Database, and Microsoft Access.

The DBMS acronym is sometimes extended to indicate the underlying database model, with RDBMS for the relational, OODBMS for the object (oriented) and ORDBMS for the object–relational model. Other extensions can indicate some other characteristic, such as DDBMS for a distributed database management systems.

The functionality provided by a DBMS can vary enormously. The core functionality is the storage, retrieval and update of data. Codd proposed the following functions and services a fully-fledged general purpose DBMS should provide: [25]

  • Data storage, retrieval and update
  • User accessible catalog or data dictionary describing the metadata
  • Support for transactions and concurrency
  • Facilities for recovering the database should it become damaged
  • Support for authorization of access and update of data
  • Access support from remote locations
  • Enforcing constraints to ensure data in the database abides by certain rules

It is also generally to be expected the DBMS will provide a set of utilities for such purposes as may be necessary to administer the database effectively, including import, export, monitoring, defragmentation and analysis utilities. [26] The core part of the DBMS interacting between the database and the application interface sometimes referred to as the database engine.

Often DBMSs will have configuration parameters that can be statically and dynamically tuned, for example the maximum amount of main memory on a server the database can use. The trend is to minimize the amount of manual configuration, and for cases such as embedded databases the need to target zero-administration is paramount.

The large major enterprise DBMSs have tended to increase in size and functionality and can have involved thousands of human years of development effort through their lifetime. [a]

Early multi-user DBMS typically only allowed for the application to reside on the same computer with access via terminals or terminal emulation software. The client–server architecture was a development where the application resided on a client desktop and the database on a server allowing the processing to be distributed. This evolved into a multitier architecture incorporating application servers and web servers with the end user interface via a web browser with the database only directly connected to the adjacent tier. [27]

A general-purpose DBMS will provide public application programming interfaces (API) and optionally a processor for database languages such as SQL to allow applications to be written to interact with the database. A special purpose DBMS may use a private API and be specifically customized and linked to a single application. For example, an email system performing many of the functions of a general-purpose DBMS such as message insertion, message deletion, attachment handling, blocklist lookup, associating messages an email address and so forth however these functions are limited to what is required to handle email.

External interaction with the database will be via an application program that interfaces with the DBMS. [28] This can range from a database tool that allows users to execute SQL queries textually or graphically, to a web site that happens to use a database to store and search information.

Application program interface

A programmer will code interactions to the database (sometimes referred to as a datasource) via an application program interface (API) or via a database language. The particular API or language chosen will need to be supported by DBMS, possible indirectly via a preprocessor or a bridging API. Some API's aim to be database independent, ODBC being a commonly known example. Other common API's include JDBC and ADO.NET.

Database languages are special-purpose languages, which allow one or more of the following tasks, sometimes distinguished as sublanguages:

    (DCL) – controls access to data (DDL) – defines data types such as creating, altering, or dropping tables and the relationships among them (DML) – performs tasks such as inserting, updating, or deleting data occurrences (DQL) – allows searching for information and computing derived information.

Database languages are specific to a particular data model. Notable examples include:

  • SQL combines the roles of data definition, data manipulation, and query in a single language. It was one of the first commercial languages for the relational model, although it departs in some respects from the relational model as described by Codd (for example, the rows and columns of a table can be ordered). SQL became a standard of the American National Standards Institute (ANSI) in 1986, and of the International Organization for Standardization (ISO) in 1987. The standards have been regularly enhanced since and is supported (with varying degrees of conformance) by all mainstream commercial relational DBMSs. [29][30] is an object model language standard (from the Object Data Management Group). It has influenced the design of some of the newer query languages like JDOQL and EJB QL. is a standard XML query language implemented by XML database systems such as MarkLogic and eXist, by relational databases with XML capability such as Oracle and DB2, and also by in-memory XML processors such as Saxon. combines XQuery with SQL. [31]

A database language may also incorporate features like:

  • DBMS-specific configuration and storage engine management
  • Computations to modify query results, like counting, summing, averaging, sorting, grouping, and cross-referencing
  • Constraint enforcement (e.g. in an automotive database, only allowing one engine type per car)
  • Application programming interface version of the query language, for programmer convenience

Database storage is the container of the physical materialization of a database. It comprises the internal (physical) level in the database architecture. It also contains all the information needed (e.g., metadata, "data about the data", and internal data structures) to reconstruct the conceptual level and external level from the internal level when needed. Putting data into permanent storage is generally the responsibility of the database engine a.k.a. "storage engine". Though typically accessed by a DBMS through the underlying operating system (and often using the operating systems' file systems as intermediates for storage layout), storage properties and configuration setting are extremely important for the efficient operation of the DBMS, and thus are closely maintained by database administrators. A DBMS, while in operation, always has its database residing in several types of storage (e.g., memory and external storage). The database data and the additional needed information, possibly in very large amounts, are coded into bits. Data typically reside in the storage in structures that look completely different from the way the data look in the conceptual and external levels, but in ways that attempt to optimize (the best possible) these levels' reconstruction when needed by users and programs, as well as for computing additional types of needed information from the data (e.g., when querying the database).

Some DBMSs support specifying which character encoding was used to store data, so multiple encodings can be used in the same database.

Various low-level database storage structures are used by the storage engine to serialize the data model so it can be written to the medium of choice. Techniques such as indexing may be used to improve performance. Conventional storage is row-oriented, but there are also column-oriented and correlation databases.

Materialized views

Often storage redundancy is employed to increase performance. A common example is storing materialized views, which consist of frequently needed external views or query results. Storing such views saves the expensive computing of them each time they are needed. The downsides of materialized views are the overhead incurred when updating them to keep them synchronized with their original updated database data, and the cost of storage redundancy.


Occasionally a database employs storage redundancy by database objects replication (with one or more copies) to increase data availability (both to improve performance of simultaneous multiple end-user accesses to a same database object, and to provide resiliency in a case of partial failure of a distributed database). Updates of a replicated object need to be synchronized across the object copies. In many cases, the entire database is replicated.

Database security deals with all various aspects of protecting the database content, its owners, and its users. It ranges from protection from intentional unauthorized database uses to unintentional database accesses by unauthorized entities (e.g., a person or a computer program).

Database access control deals with controlling who (a person or a certain computer program) is allowed to access what information in the database. The information may comprise specific database objects (e.g., record types, specific records, data structures), certain computations over certain objects (e.g., query types, or specific queries), or using specific access paths to the former (e.g., using specific indexes or other data structures to access information). Database access controls are set by special authorized (by the database owner) personnel that uses dedicated protected security DBMS interfaces.

This may be managed directly on an individual basis, or by the assignment of individuals and privileges to groups, or (in the most elaborate models) through the assignment of individuals and groups to roles which are then granted entitlements. Data security prevents unauthorized users from viewing or updating the database. Using passwords, users are allowed access to the entire database or subsets of it called "subschemas". For example, an employee database can contain all the data about an individual employee, but one group of users may be authorized to view only payroll data, while others are allowed access to only work history and medical data. If the DBMS provides a way to interactively enter and update the database, as well as interrogate it, this capability allows for managing personal databases.

Data security in general deals with protecting specific chunks of data, both physically (i.e., from corruption, or destruction, or removal e.g., see physical security), or the interpretation of them, or parts of them to meaningful information (e.g., by looking at the strings of bits that they comprise, concluding specific valid credit-card numbers e.g., see data encryption).

Change and access logging records who accessed which attributes, what was changed, and when it was changed. Logging services allow for a forensic database audit later by keeping a record of access occurrences and changes. Sometimes application-level code is used to record changes rather than leaving this to the database. Monitoring can be set up to attempt to detect security breaches.

Database transactions can be used to introduce some level of fault tolerance and data integrity after recovery from a crash. A database transaction is a unit of work, typically encapsulating a number of operations over a database (e.g., reading a database object, writing, acquiring lock, etc.), an abstraction supported in database and also other systems. Each transaction has well defined boundaries in terms of which program/code executions are included in that transaction (determined by the transaction's programmer via special transaction commands).

The acronym ACID describes some ideal properties of a database transaction: atomicity, consistency, isolation, and durability.

A database built with one DBMS is not portable to another DBMS (i.e., the other DBMS cannot run it). However, in some situations, it is desirable to migrate a database from one DBMS to another. The reasons are primarily economical (different DBMSs may have different total costs of ownership or TCOs), functional, and operational (different DBMSs may have different capabilities). The migration involves the database's transformation from one DBMS type to another. The transformation should maintain (if possible) the database related application (i.e., all related application programs) intact. Thus, the database's conceptual and external architectural levels should be maintained in the transformation. It may be desired that also some aspects of the architecture internal level are maintained. A complex or large database migration may be a complicated and costly (one-time) project by itself, which should be factored into the decision to migrate. This in spite of the fact that tools may exist to help migration between specific DBMSs. Typically, a DBMS vendor provides tools to help importing databases from other popular DBMSs.

After designing a database for an application, the next stage is building the database. Typically, an appropriate general-purpose DBMS can be selected to be used for this purpose. A DBMS provides the needed user interfaces to be used by database administrators to define the needed application's data structures within the DBMS's respective data model. Other user interfaces are used to select needed DBMS parameters (like security related, storage allocation parameters, etc.).

When the database is ready (all its data structures and other needed components are defined), it is typically populated with initial application's data (database initialization, which is typically a distinct project in many cases using specialized DBMS interfaces that support bulk insertion) before making it operational. In some cases, the database becomes operational while empty of application data, and data are accumulated during its operation.

After the database is created, initialized and populated it needs to be maintained. Various database parameters may need changing and the database may need to be tuned (tuning) for better performance application's data structures may be changed or added, new related application programs may be written to add to the application's functionality, etc.

Sometimes it is desired to bring a database back to a previous state (for many reasons, e.g., cases when the database is found corrupted due to a software error, or if it has been updated with erroneous data). To achieve this, a backup operation is done occasionally or continuously, where each desired database state (i.e., the values of its data and their embedding in database's data structures) is kept within dedicated backup files (many techniques exist to do this effectively). When it is decided by a database administrator to bring the database back to this state (e.g., by specifying this state by a desired point in time when the database was in this state), these files are used to restore that state.

Static analysis techniques for software verification can be applied also in the scenario of query languages. In particular, the *Abstract interpretation framework has been extended to the field of query languages for relational databases as a way to support sound approximation techniques. [32] The semantics of query languages can be tuned according to suitable abstractions of the concrete domain of data. The abstraction of relational database system has many interesting applications, in particular, for security purposes, such as fine grained access control, watermarking, etc.

Other DBMS features might include:

    – This helps in keeping a history of the executed functions.
  • Graphics component for producing graphs and charts, especially in a data warehouse system. – Performs query optimization on every query to choose an efficient query plan (a partial order (tree) of operations) to be executed to compute the query result. May be specific to a particular storage engine.
  • Tools or hooks for database design, application programming, application program maintenance, database performance analysis and monitoring, database configuration monitoring, DBMS hardware configuration (a DBMS and related database may span computers, networks, and storage units) and related database mapping (especially for a distributed DBMS), storage allocation and database layout monitoring, storage migration, etc.

Increasingly, there are calls for a single system that incorporates all of these core functionalities into the same build, test, and deployment framework for database management and source control. Borrowing from other developments in the software industry, some market such offerings as "DevOps for database". [33]

The first task of a database designer is to produce a conceptual data model that reflects the structure of the information to be held in the database. A common approach to this is to develop an entity–relationship model, often with the aid of drawing tools. Another popular approach is the Unified Modeling Language. A successful data model will accurately reflect the possible state of the external world being modeled: for example, if people can have more than one phone number, it will allow this information to be captured. Designing a good conceptual data model requires a good understanding of the application domain it typically involves asking deep questions about the things of interest to an organization, like "can a customer also be a supplier?", or "if a product is sold with two different forms of packaging, are those the same product or different products?", or "if a plane flies from New York to Dubai via Frankfurt, is that one flight or two (or maybe even three)?". The answers to these questions establish definitions of the terminology used for entities (customers, products, flights, flight segments) and their relationships and attributes.

Producing the conceptual data model sometimes involves input from business processes, or the analysis of workflow in the organization. This can help to establish what information is needed in the database, and what can be left out. For example, it can help when deciding whether the database needs to hold historic data as well as current data.

Having produced a conceptual data model that users are happy with, the next stage is to translate this into a schema that implements the relevant data structures within the database. This process is often called logical database design, and the output is a logical data model expressed in the form of a schema. Whereas the conceptual data model is (in theory at least) independent of the choice of database technology, the logical data model will be expressed in terms of a particular database model supported by the chosen DBMS. (The terms data model and database model are often used interchangeably, but in this article we use data model for the design of a specific database, and database model for the modeling notation used to express that design).

The most popular database model for general-purpose databases is the relational model, or more precisely, the relational model as represented by the SQL language. The process of creating a logical database design using this model uses a methodical approach known as normalization. The goal of normalization is to ensure that each elementary "fact" is only recorded in one place, so that insertions, updates, and deletions automatically maintain consistency.

The final stage of database design is to make the decisions that affect performance, scalability, recovery, security, and the like, which depend on the particular DBMS. This is often called physical database design, and the output is the physical data model. A key goal during this stage is data independence, meaning that the decisions made for performance optimization purposes should be invisible to end-users and applications. There are two types of data independence: Physical data independence and logical data independence. Physical design is driven mainly by performance requirements, and requires a good knowledge of the expected workload and access patterns, and a deep understanding of the features offered by the chosen DBMS.

Another aspect of physical database design is security. It involves both defining access control to database objects as well as defining security levels and methods for the data itself.


A database model is a type of data model that determines the logical structure of a database and fundamentally determines in which manner data can be stored, organized, and manipulated. The most popular example of a database model is the relational model (or the SQL approximation of relational), which uses a table-based format.

Feature binning is only supported for point and multipoint feature classes stored in an enterprise geodatabase or database. The data cannot be versioned or archive enabled.

Use the same coordinate system for the bins as the map containing the binned feature layer to avoid dynamic (on-the-fly) projection. If you are unsure of the coordinate system to use, an equal area projection such as World Cylindrical Equal Area is recommended. You cannot use a custom coordinate system.

For very large datasets or if the data is not updated often, you can enable a static cache of aggregated results. However, the cache is not necessarily created for all levels of detail. The static cache can be managed by running the Manage Feature Bin Cache tool. Use this tool to specify the levels of detail of the static cache.

A bin-enabled layer switches to dynamic mode in a map when you zoom past the level of detail extent of the static cache. The Max cached level property on a bin-enabled layer's Layer Properties dialog box lists the maximum level of detail of the static cache. Static caches are generated using all the features of the dataset. So, if you use a definition query, or apply a time or range filter on a bin-enabled feature layer in a map, the static cache is ignored and the bin aggregation occurs dynamically.

Use the Disable Feature Binning tool to disable the binning capability from a layer if necessary. You can also turn off bin drawing for a layer in a map or scene without disabling feature binning.

When feature binning is dynamic and you want to update the list of summary statistics stored in the feature class, you must disable and reenable feature binning. You can add new summary statistics to the feature layer in a map or scene from the layer's Summary Statistics dialog box. These summary statistics are stored with the layer only. They are not stored in the source feature class.

Format column length in SSMS output

SQL Server 2012. Sample query at the bottom of this post.

I'm trying to create a simple report for when a given database was last backed up.

When executing the sample query with output to text in SSMS, the DB_NAME column is formatted to be the max possible size for data (same issue exists in DB2, btw). So, I've got a column that contains data that is never more than, say, 12 characters, but it's stored in a varchar(128) , I get 128 characters of data no matter what. RTRIM has no effect on the output.

Is there an elegant way that you know of to make the formatted column length be the max size of actual data there, rather than the max potential size of data?

I guess there exists an xp_sprintf() function, but I'm not familiar with it, and it doesn't look terribly robust.

I've tried casting it like this:

But then SQL Server won't let me use the variable @database_name_Length in my varchar definition when casting. SQL Server, apparently, demands a literal number when declaring the char or varchar variable.

I'm down to building the statement in a string and using something like sp_executesql , or building a temp table with the actual column lengths I need, both of which are really a bit more trouble than I was hoping to go to just to NOT get 100 spaces in my output on a 128 character column.

Have searched the interwebs and found bupkus.

Maybe I'm searching for the wrong thing, or Google is cross with me.

It seems that SSMS will format the column to be the maximum size allowed, even if the actual data is much smaller. I was hoping for an elegant way to "fix" this without jumping through hoops. I'm using SSMS 2012.

If I go to Results To Grid and then to Excel or something similar, the trailing space is eliminated. I was hoping to basically create a report that I email, though.